Overview of data quality challenges in the context of Big Data

Abstract

Data quality management systems are thoroughly researched topics and have resulted in many tools and techniques developed by both academia and industry. However, the advent of Big Data might pose some serious questions pertaining to the applicability of existing data quality concepts. There is a debate concerning the importance of data quality for Big Data; one school of thought argues that high data quality methods are essential for deriving higher level analytics while another school of thought argues that data quality level will not be so important as the volume of Big Data would be used to produce patterns and some amount of dirty data will not mask the analytic results which might be derived. This paper aims to investigate various components and activities forming part of data quality management such as dimensions, metrics, data quality rules, data profiling and data cleansing. The result list existing challenges and future research areas associated with Big Data for data quality management.