Problems and systematic solutions in data quality

Abstract

Data are important for making decisions. However, the quality of the data affects the quality of decisions. Data mining as one of the most important sources of knowledge needs high quality data to mine, but data of sufficient quality is often lacking. By systematically analysing the reasons causing low data quality in data mining, we found that general methods on improving data quality by data cleaning are not enough. A new method for improving data quality called data mining consulting has been established. It defines data quality in a wider range from the customer view of data mining, finds potential data quality problems at an earlier stage and solves the data quality problem by a series methods including software techniques, data mining principles and management rules such as ISO 9000. Its application in a web company shows that it is practical and can increase data quality from the very beginning.