From the edge to the cloud: A continuous delivery and preparation model for processing big IoT data

Abstract

This paper presents a processing model for big IoT data. The model includes a continuous delivery scheme based on building blocks for constructing software pipelines from the edge to the cloud. It also includes a data preparation scheme based on parallel patterns for establishing, in an efficient manner, controls over the production and consumption of IoT data. This scheme adds data properties such as cost-efficiency storage, security, and reliability, which are useful to avoid alterations in data and repudiation situations as well as to mitigate risks still arisen in the cloud such as confidentiality violations and service outages. An overlay structure, including planes such as pub/sub, control, and preservation, integrates the proposed schemes into software pipelines. The proposed model was developed in both prototype and simulator of software pipelines. Case studies were conducted based on pipeline services deployed from the edge, passing from the fog to the cloud for processing and managing real climate data repositories, which were produced by three different data sensor sources, such as ground stations deployed on Mexico and Spain, as well as small distributed IoT devices. Information sharing patterns for end-users to retrieve raw and/or processed IoT data were also studied. The experimental evaluation revealed the feasibility of using continuous delivery scheme to create dataflows from the edge to the cloud, the efficacy of the overlay structure to create information sharing patterns, as well as the efficiency of data preparation schemes and parallel patterns to improve the end-user service experience in comparison with traditional state-of-the-art solutions.