Outlier detection from multiple data sources

Abstract

The preparation of a dataset by merging multiple data sources using the data fusion method may lead to the loss of vital information from each multi-source dataset and a certain amount of correlative information among the multiple data sources. Based on the extensive analysis of “the unique characteristics” of multi-source outliers, we propose multi-source outlier detection techniques to reliably identify outliers in multiple datasets. Several real-world examples are considered to classify multi-source outliers into three types (Type I-III) depending on the correlation among datasets. We design a baseline algorithm, which is an intuitive solution, and an optimal algorithm known as multiple-data-sources oriented outlier detection (MOD) to obtain high-score outliers. In addition, we build the MOD+ method to speed up the outlier detection process. A new density metric combining kNN and RNN is introduced to evaluate the deviation degrees of multi-source outliers. The new outlier are applied to develop three outlier-join operators. MOD and MOD+ are adept at (1) mining outlier information from each one of multi-source datasets and (2) sensing correlative outlier information among these multiple datasets. We implement and evaluate the three outlier detection algorithms by using synthetic and real-world datasets. The experimental results demonstrate that the proposed methods are promising and practical in the context of detecting outliers from multi-source datasets.