Finding Interesting Cleaning Rules from Dirty Data

Abstract

Data cleaning is an important problem and data quality rules are the most promising way to address it. Previous researches have focused on constraints, such as functional dependencies (FDs), conditional functional dependencies (CFDs), fixing rules, and editing rules. These rules can be provided by experts, or discovered by mining algorithms and then experts verify. But in the condition of dirty dataset, it is highly unlikely that useful FDs could be discovered. If there are no useful cleaning rules, the user can not capture and correct errors in data. In this paper, we develop a system that is able to find interesting rules from dirty data. There is no hope to mine significant functional dependencies directly by the TANE algorithm because of the existence of errors in data. Thus, we discover approximate functional dependencies as candidate cleaning rules with the improved TANE algorithm. The number of candidate rules is very large, making it expensive for the user verification. Since not all candidate rules make sense in practice, we design two kinds of ranking methods: the support-based ranking and the comprehensive score ordering. Our ranking methods usually put the most useful rules ahead. Experimental results show that our sorting methods can effectively help the user find useful cleaning rules.