Pattern-based serverless data processing pipelines for Function-as-a-Service orchestration systems

Abstract

The serverless cloud computing execution model presents an opportunity for implementing data processing pipelines in a cost efficient manner with minimum or no consideration about managing their operational aspects. In doing so, however, a major concern that emerges is the adoption of provider-specific languages required for orchestrating serverless functions and supporting services into the higher order logic of pipelines. As a result, the vendor lock-in problem inherent to cloud computing becomes further intensified. In this work we propose an approach to mitigate this issue through the adoption and adaptation of well-known patterns from the literature as the means for modeling the pipeline logic. By providing mappings from these patterns to vendor-specific orchestration language constructs we allow for efficient realization of provider-agnostic pipeline models as provider-specific executable workflows. An industrial case study provides evidence towards the suitability of our proposal for practical purposes.