A Cost-Effective Framework for Running Industrial Big Data Analysis Applications in Public Clouds

Abstract

Nowadays, the improvement of the data acquisition capability of the Industrial Internet of Things (IIoT) systems has brought about higher data throughput. Users can deploy industrial big data analysis applications on cloud platforms in a pay-as-you-go way to deal with the unstable data generation and data analysis workloads in the IIoT scenario. To procure stable and flexible computing resources, as a traditional pay-as-you-go cloud instance procurement option, on-demand instances are widely used, but their expensive prices also significantly increase users’ cost burden. To reduce the data analysis cost, in this article, we establish a per-job cost-effective framework adopting spot instances, on-demand instances, and cloud storage for industrial big data analysis applications. In our framework, we take advantage of spot instances, which are computing instances provided at low prices under a pay-as-you-go model, to achieve cost savings. However, using spot instances carries the risk of being interrupted. Therefore, we propose to use a checkpointing mechanism to back up intermediate results to cloud storage to reduce the potential loss caused by spot instance interruptions. Considering the time sensitivity of industrial big data analysis applications, we use on-demand instances as alternative computing resources after spot instance interruptions to ensure that users’ jobs can be completed without high time latencies. Evaluation results show that our framework can achieve cost savings as well as minimize time latencies for users’ jobs.