An edge-aware autonomic runtime for data streaming and in-transit processing

Abstract

One of the major endeavors of modern cyberinfrastructure (CI) is to carry content produced on remote data sources, such as sensors and scientific instruments, and to deliver it to end users and workflow applications. Maintaining data quality, data resolution, and on-time data delivery and considering the increasing number of computing, storage, and network resources are challenging tasks that require a receptive system able to adapt to ever-changing demands. In this paper, we propose a mathematical model of such system by expressing the dynamic stages of different resources in the context of edge and in-transit computing. By considering resource utilization, approximation techniques, and user constraints, our proposed model generates mappings of different workflow stages on heterogeneous geographically distributed resources. Specifically, we propose an autonomic runtime management layer that adapts the data resolution being delivered to the users by implementing feedback loops over the resources involved in the delivery and processing of data streams. The implementation of our model is based on a subscription-based data streaming framework that enables the integration of large facilities and advanced CI. Moreover, the idea of stream or request aggregation is incorporated into our framework, which eliminates redundant data streams to save bandwidth. Experimental results show that dynamically adapting data resolution and stream aggregation can overcome bandwidth limitations in wide-area streaming analytics by leveraging the resources at the edge and in-transit.