An empirical analysis of stateful operator migration for online scheduling in distributed stream processing systems

Abstract

Streaming data is the data from sensors as well as other real-time surveillance systems. Distributed stream processing systems are the software that manages such data. Such frameworks have to deliver outcomes on the go instantly. They are susceptible to delay and malfunction or system failures. The system must be tolerant of faults and always accessible. Many variables, such as improved network arrival rates, node failures, and so on, disrupt the system's reliability. Some operators need to be relocated online from one physical resource to another to manage or reimburse a slow or failing node. In this study, we propose a co-location based systematic migration heuristic for live operator migration between physical resources using a migration map revised with costs for each migration. The suggested method evaluates continuous operator performance patterns and makes online scheduling decisions based on the same. The decisions include migrating operators during a node failure or straggling.