An External Data Integrity Tracking and Verification System for Universal Stream Computing System Framework

Abstract

With the development of the cloud computing and big data, massive data based computing and storage have emerged as core technologies. Moreover, a key problem to ensure data availability and analysis accuracy is the data integrity which has received extensive attention. Compared to the mature solution in the batch computing, the data integrity in the stream computing did not receive due attention. However, the characters of timeliness, volatility, suddenness and disorder of big data make the data integrity in the stream computing a very difficult problem. Based on the framework of current general stream computing system, in this paper, we construct an externally data integrity tracking and verification system for the real-time tracking and analysing of the data content and processing path of each message, and the discovering of the data integrity problem. We use the common cryptographic algorithm to construct a universal data integrity tracking and verification scheme outside the stream computing system to ensure the integrity and consistency of the data. Then we give the formal security analysis in standard model, and perform simulations in cloud environment. Results show that the proposed system can verify data integrity efficiently without compromising the efficiency.