A nodes scheduling model based on Markov chain prediction for big streaming data analysis

Abstract

SummaryStreaming data analysis is an important part of big data processing. However, streaming data is difficult to be analyzed and processed in real time because of the rapid data arriving speed and huge size of data set in stream model. The paper proposes a nodes scheduling model based on Markov chain prediction for analyzing big streaming data in real time by following three steps: (i) construct data state transition graph using Markov chain to predict the varying trend of big streaming data; (ii) choose appropriate cloud computing nodes to process big streaming data depending on the predicted result of the data state transition graph; and (iii) assign big streaming data to these computing nodes using the load balancing theory, which ensures that all subtasks are accomplished synchronously. Experiments demonstrate that the proposed scheduling algorithm can fast process big streaming data effectively. Copyright © 2014 John Wiley & Sons, Ltd.