Error Detection and Uncertainty Modeling for Imprecise Data

Abstract

In this paper, we propose a method to derive and model data uncertainty from imprecise data. We view data imprecision and errors as the outcome of the precise data exposed to some uncertain channels, and our scheme is to directly derive the data uncertainty model from imprecise data, such that the derived data uncertainty information may be integrated into the succeeding mining process. To achieve the goal, we propose an expectation maximization (EM) based approach to detect erroneous data entries from the input data. The data uncertainty models are constructed by applying statistical analysis to the detected errors. Experimental results show that the proposed error detection approach can locate data errors and suggest alternative data entry values to improve classifiers built from imprecise data. In addition, the uncertain models derived for each individual attributes are shown to be close to the genuine uncertainty models used to corrupt the data.