The Challenges of Data Quality and Data Quality Assessment in the Big Data Era

Li Cai,Yangyong Zhu

doi:10.5334/dsj-2015-002

Abstract

High-quality data are the precondition for analyzing and using big data and for guaranteeing the value of the data. Currently, comprehensive analysis and research of quality standards and quality assessment methods for big data are lacking. First, this paper summarizes reviews of data quality research. Second, this paper analyzes the data characteristics of the big data environment, presents quality challenges faced by big data, and formulates a hierarchical data quality framework from the perspective of data users. This framework consists of big data quality dimensions, quality characteristics, and quality indexes. Finally, on the basis of this framework, this paper constructs a dynamic assessment process for data quality. This process has good expansibility and adaptability and can meet the needs of big data quality assessment. The research results enrich the theoretical scope of big data and lay a solid foundation for the future by establishing an assessment model and studying evaluation algorithms.

Highlights

Many significant technological changes have occurred in the information technology industry since the beginning of the 21st century, such as cloud computing, the Internet of Things, and social networking
How to ensure big data quality and how to analyze and mine information and knowledge hidden behind the data become major issues for industry and academia
We analyzed the challenges faced by big data quality and proposed the establishment and hierarchical structure of a data quality framework

Summary

Introduction

Many significant technological changes have occurred in the information technology industry since the beginning of the 21st century, such as cloud computing, the Internet of Things, and social networking. The development of these technologies has made the amount of data increase continuously and accumulate at an unprecedented speed. By rapidly acquiring and analyzing big data from various sources and with various uses, researchers and decision-makers have gradually realized that this massive amount of information has benefits for understanding customer needs, improving service quality, and predicting and preventing risks. Art. 2, page 2 of 10 Cai and Zhu: The Challenges of Data Quality and Data Quality Assessment in the Big Data Era

Literature Review on Data Quality

The challenges of data quality

Quality Criteria of Big Data

QUALITY ASSESSMENT PROCESS FOR BIG DATA

Findings

Conclusion