Dynamic Workload Management for Large-Scale Data Processing Using Serverless Architectures on Google Cloud Platform

Friedrich Schneider Martin

Dynamic Workload Management for Large-Scale Data Processing Using Serverless Architectures on Google Cloud Platform

Authors

Friedrich Schneider Martin

Cloud Data Engineer – Serverless & Scalable Workload Automation on GCP, Spain

Keywords:

Serverless computing, workload management, Google Cloud Platform, large-scale data processing, cloud functions, data pipelines, elastic computing

Synopsis

The demand for scalable, cost-efficient, and elastic data processing frameworks has intensified with the exponential growth of big data workloads. Traditional server-based architectures struggle to adapt to fluctuating workloads due to over-provisioning and underutilization. Serverless computing offers an on-demand execution model, significantly optimizing resource utilization and cost. This paper proposes a dynamic workload management framework for large-scale data processing using serverless architectures on the Google Cloud Platform (GCP). By leveraging tools like Cloud Functions, Cloud Run, and Dataflow, we demonstrate the orchestration of complex workflows with reduced latency and increased throughput. Our experimental analysis illustrates improved performance metrics compared to baseline VM-based architectures. Results indicate that serverless platforms are ideal for unpredictable data volumes and real-time processing needs.

References

[1] Baldini, I., Castro, P., Chang, K., et al. (2017). Serverless computing: Current trends and open problems. IEEE Internet Computing, 21(5), 52–62.

[2] Gummadi, V. P. K. (2019). Microservices architecture with APIs: Design, implementation, and MuleSoft integration. Journal of Electrical Systems, 15(4), 130–134. https://doi.org/10.52783/jes.9328

[3] Jonas, E., Schleier-Smith, J., Sreekanti, V., et al. (2019). Cloud programming simplified: A Berkeley view on serverless computing. Communications of the ACM, 62(6), 54–62.

[4] Hellerstein, J. M., Gonzalez, J. E., et al. (2018). Serverless computing for data science workflows. IEEE Data Engineering Bulletin, 41(4), 50–61.

[5] Wang, L., Zhang, M., Ristenpart, T., et al. (2020). Peeking behind the curtains of serverless platforms. Proceedings of the USENIX Annual Technical Conference, 27(1), 133–146.

[6] Gummadi, V. P. K. (2026). Infrastructure optimization techniques for enterprise integration platforms: A comprehensive analysis. Computer Fraud & Security, 2026(1), 37–44. https://doi.org/10.52710/cfs.875

[7] Spillner, J., Mateos, C., and Monge, D. A. (2018). Faaster, better, cheaper: The prospect of serverless scientific computing and HPC. Future Generation Computer Systems, 85(1), 50–58.

[8] Abad, C. L., Michael, C., and Pascual, F. (2021). Toward real-time analytics using serverless computing. Journal of Cloud Computing, 10(1), 20–34.

[9] McGrath, G., & Brenner, P. R. (2017). Serverless computing: Design, implementation, and performance. 2017 IEEE International Conference on Cloud Engineering, 3(1), 33–40.

[10] Gummadi, V. P. K. (2023). MuleSoft batch processing: High-volume streaming architecture. Computer Fraud & Security, 2023(12), 50–57. https://doi.org/10.52710/cfs.886

[11] Roberts, M. (2018). Serverless architectures. IEEE Software, 35(2), 96–101.

[12] Duck, G., & Thomas, S. (2020). Asynchronous data pipelines with GCP. ACM SIGMOD Record, 49(1), 14–19.

[13] Oakes, E., Yang, L., Zhou, K., et al. (2018). SOCK: Rapid task provisioning with serverless-optimized containers. USENIX ATC, 38(2), 57–69.

[14] Adzic, G., & Chatley, R. (2017). Serverless computing: Economic and architectural impact. IEEE Software, 34(3), 76–81.

[15] Malawski, M., Figiela, K., et al. (2017). Benchmarking AWS Lambda and Google Cloud Functions. Future Generation Computer Systems, 81(2), 142–153.

[16] Gummad, V. P. K. (2025). Flex gateway, service mesh, and advanced API management evolution. International Journal of Applied Mathematics, 38(9s), 2199–2206. https://doi.org/10.12732/ijam.v38i9s.1643

[17] Lloyd, W., Ramesh, S., Chinthalapati, S., et al. (2018). Serverless computing: An investigation of factors influencing microservice performance. IC2E, 24(1), 59–68.

[18] Castro, P., Ishakian, V., et al. (2017). The serverless trilemma: Function composition for serverless computing. Proceedings of the ACM Symposium on Cloud Computing, 12(3), 1–13.

[19] Xu, X., Li, X., et al. (2021). Enhancing serverless platforms for event-driven computing. IEEE Transactions on Cloud Computing, 9(2), 405–418.