Heterogeneous Data Integration and Fusion System Based on Metadata Conflict Algorithms in USPIOT

Abstract

With the development of the Internet of Things, data islands and information redundancy which caused by massive heterogeneous data has become a challenge of many commercial applications and scientific research hot spots. First, we proposed and designed a MapReduce data integration and data fusion architecture, then, we established a stable information processing environment and a unified information retrieval interface for users, and then we analyzed MapReduce implementation processes, management processes, and the connection process. Meanwhile we proposed a conflict processing algorithms in processing heterogeneous redundant data problems by using metadata, and applied them to the processing of creating the virtual database using MapReduce technology, eliminating the redundant information to enhance the utilization of data space. Finally, we established the bench system to provide a unified heterogeneous data programming interface and memory management with heterogeneous data integration and fusion for users in our developed platform USPIOT (Ubiquitous Service Platform for Internet Of Things) and achieved better results.