Continuous Dataflow Update Strategies for Mission-Critical Applications

Abstract

Continuous data flows complement scientific work-flows by allowing composition of real time data ingest and analytics pipelines to process data streams from pervasive sensors and always-on scientific instruments. Such data flows are mission-critical applications that cannot suffer downtime, need to operate consistently, and are long running, but may need to be updated to fix bugs or add features. This poses the problem: How do we update the continuous dataflow application with minimal disruption? In this paper, we formalize different types of dataflow update models for continuous dataflow applications, and identify the qualitative and quantitative metrics to be considered when choosing an update strategy. We propose five dataflow update strategies, and analytically characterize their performance trade-offs. We validate one of these consistent, low-latency update strategies using the Floe dataflow engine for an eEngineering application from the Smart Power Grid domain, and show its relative performance benefits against a naive update strategy.