Dirty Datasets Research Articles

Elucidating robust learning with uncertainty-aware corruption pattern estimation

Robust learning methods aim to learn a clean target distribution from noisy and corrupted training data where a specific corruption pattern is often assumed a priori. Our proposed method can not only successfully learn the clean target distribution from a dirty dataset but also can estimate the underlying noise pattern. To this end, we leverage a mixture-of-experts model that can distinguish two different types of predictive uncertainty, aleatoric and epistemic uncertainty. We show that the ability to estimate the uncertainty plays a significant role in elucidating the corruption patterns as these two objectives are tightly intertwined. We also present a novel validation scheme for evaluating the performance of the corruption pattern estimation. Our proposed method is extensively assessed in terms of both robustness and corruption pattern estimation in the computer vision domain. Code has been made publicly available at https://github.com/jeongeun980906/Uncertainty-Aware-Robust-Learning.

Self-Supervised and Interpretable Data Cleaning with Sequence Generative Adversarial Networks

We study the problem of self-supervised and interpretable data cleaning, which automatically extracts interpretable data repair rules from dirty data. In this paper, we propose a novel framework, namely Garf, based on sequence generative adversarial networks (SeqGAN). One key information Garf tries to capture is data repair rules (for example, if the city is "Dothan", then the county should be "Houston"). Garf employs a SeqGAN consisting of a generator G and a discriminator D that trains G to learn the dependency relationships ( e.g. , given a city value "Dothan" as input, the county can be determined as "Houston"). After training, the generator G can be used to generate data repair rules, but may contain both trusted and untrusted rules, especially when learning from dirty data. To mitigate this problem, Garf further updates the learned relationships with another discriminator D' to iteratively improve the quality of both rules and data. Garf takes advantages of both logical and learning-based methods, which allow cleaning dirty data with high interpretability and have no requirements for prior knowledge and training data. Extensive experiments on real-world and synthetic datasets demonstrate the effectiveness of Garf. Garf achieves new state-of-the-art data cleaning result with high accuracy, through learning from dirty datasets without human supervision.

Unsupervised Anomaly Detection and Segmentation on Dirty Datasets

Industrial quality control is an important task. Most of the existing vision-based unsupervised industrial anomaly detection and segmentation methods require that the training set only consists of normal samples, which is difficult to ensure in practice. This paper proposes an unsupervised framework to solve the industrial anomaly detection and segmentation problem when the training set contains anomaly samples. Our framework uses a model pretrained on ImageNet as a feature extractor to extract patch-level features. After that, we propose a trimming method to estimate a robust Gaussian distribution based on the patch features at each position. Then, with an iterative filtering process, we can iteratively filter out the anomaly samples in the training set and re-estimate the Gaussian distribution at each position. In the prediction phase, the Mahalanobis distance between a patch feature vector and the center of the Gaussian distribution at the corresponding position is used as the anomaly score of this patch. The subsequent anomaly region segmentation is performed based on the patch anomaly score. We tested the proposed method on three datasets containing the anomaly samples and obtained state-of-the-art performance.

Progressive Entity Matching via Cost Benefit Analysis

Entity matching (EM) is a fundamental problem in data preprocessing, and is a long running topic in big data analytics and mining communities. In big data era, (nearly) real-time data applications become popular, and call for progressive EM, which produces as many match pairs as possible in very limited time. Previous progressive EM focus on memory based solutions, but disk based solutions are necessary when dirty datasets cannot be fully loaded into memory. To this end, we propose a cost benefit analysis based progressive EM approach, which partitions data according to coarse clustering results and then iteratively schedules data partitions in a greedy way for high progressive resolution. At first, based on estimated record pair similarities, records are fast coarsely clustered; then, record clusters with near average similarities are greedily distributed to the same partitions, and data partitions are cached. After that, cost model is defined with time and space constrains, and benefit model is defined with expected resolution results. On the basis of the cost benefit model, a greedy approximate method is proposed to effectively schedule data for high progressiveness of EM. Finally, we implement extensive experiments over several datasets to evaluate our approach, and show its advantages over existing works.

Sample Reduction for Physiological Data Analysis Using Principal Component Analysis in Artificial Neural Network

With its potential, extensive data analysis is a vital part of biomedical applications and of medical practitioner interpretations, as data analysis ensures the integrity of multidimensional datasets and improves classification accuracy; however, with machine learning, the integrity of the sources is compromised when the acquired data pose a significant threat in diagnosing and analysing such information, such as by including noisy and biased samples in the multidimensional datasets. Removing noisy samples in dirty datasets is integral to and crucial in biomedical applications, such as the classification and prediction problems using artificial neural networks (ANNs) in the body’s physiological signal analysis. In this study, we developed a methodology to identify and remove noisy data from a dataset before addressing the classification problem of an artificial neural network (ANN) by proposing the use of the principal component analysis–sample reduction process (PCA–SRP) to improve its performance as a data-cleaning agent. We first discuss the theoretical background to this data-cleansing methodology in the classification problem of an artificial neural network (ANN). Then, we discuss how the PCA is used in data-cleansing techniques through a sample reduction process (SRP) using various publicly available biomedical datasets with different samples and feature sizes. Lastly, the cleaned datasets were tested through the following: PCA–SRP in ANN accuracy comparison testing, sensitivity vs. specificity testing, receiver operating characteristic (ROC) curve testing, and accuracy vs. additional random sample testing. The results show a significant improvement in the classification of ANNs using the developed methodology and suggested a recommended range of selectivity (Sc) factors for typical cleaning and ANN applications. Our approach successfully cleaned the noisy biomedical multidimensional datasets and yielded up to an 8% increase in accuracy with the aid of the Python language.

Evaluating Top-N Join Queries with Real-time Entity Resolution

Using nonmonotone ranking functions in top-N queries is a challenge. Traditional techniques for top-N queries are based on clean data without entity resolution (ER). For dirty datasets with duplicate tuples referring to the same real-world entity, these techniques may yield top-N tuples duplicates for a query. Consequently, the effective size of the result set of the query is less than N, and some useful tuples may fail to be retrieved from the datasets, which leads to poor effectiveness. Using an ER-Index based on a divide-and-conquer mechanism and nonmonotone ranking functions, in this paper, we propose a method for processing top-N join queries with real-time ER. This method integrates ER with the processing of a top-N join query over dirty datasets on the fly. Extensive experiments are conducted to measure the effectiveness and efficiency of the method over dirty datasets.

On the nesting of a contour dataset and its use in repair

ABSTRACTA contour dataset is a common source of data in biomedical imaging and GIS. In practice, many contour datasets violate key assumptions: contours self-intersect or intersect other contours, which can disrupt the performance of algorithms. We call such data dirty. Avoiding this issue, algorithms in the literature assume clean data. This paper considers the issue of dirty datasets, in the context of nesting.Contours are often nested inside other contours, and the nesting level of a contour is important, as it affects the position of the inside of the shape. This paper analyzes the nesting of a dirty dataset, then shows how to use this nesting analysis to repair the data. A theme is the power of image space algorithms in the presence of noise. Image space algorithms for polygon area and Boolean operations of polygons are provided.

An Aggressive Scheme for Identifying Replicas in Limited Time

We advise two novel, progressive duplicate recognition calculations namely progressive sorted neighborhood method (PSNM), which performs best on small , almost clean datasets, and progressive obstructing (PB), which performs best on large and incredibly dirty datasets. Duplicate recognition is the procedure of determining multiple representations of same real life organizations. Today, duplicate recognition techniques have to process ever bigger datasets in ever shorter time: maintaining the caliber of a dataset becomes more and more difficult. We present two novel, progressive duplicate recognition calculations that considerably boost the efficiency to find duplicates when the execution time is restricted: They increase the gain from the overall process inside the time available by confirming most results much sooner than traditional approaches. Comprehensive experiments reveal that our progressive calculations can double the amount efficiency with time of traditional duplicate recognition and considerably enhance related work. Progressive obstructing is really a novel approach that develops upon an equidistant obstructing technique and also the successive enlargement of blocks. Like PSNM, additionally, it presorts the

ActiveClean

Analysts often clean dirty data iteratively--cleaning some data, executing the analysis, and then cleaning more data based on the results. We explore the iterative cleaning process in the context of statistical model training, which is an increasingly popular form of data analytics. We propose ActiveClean, which allows for progressive and iterative cleaning in statistical modeling problems while preserving convergence guarantees. ActiveClean supports an important class of models called convex loss models (e.g., linear regression and SVMs), and prioritizes cleaning those records likely to affect the results. We evaluate ActiveClean on five real-world datasets UCI Adult, UCI EEG, MNIST, IMDB, and Dollars For Docs with both real and synthetic errors. The results show that our proposed optimizations can improve model accuracy by up-to 2.5x for the same amount of data cleaned. Furthermore for a fixed cleaning budget and on all real dirty datasets, ActiveClean returns more accurate models than uniform sampling and Active Learning.

A proficient cost reduction framework for de-duplication of records in data integration.

BackgroundRecord de-duplication is a process of identifying the records referring to the same entity. It has a pivotal role in data mining applications, which involves the integration of multiple data sources and data cleansing. It has been a challenging task due to its computational complexity and variations in data representations across different data sources. Blocking and windowing are the commonly used methods for reducing the number of record comparisons during record de-duplication. Both blocking and windowing require tuning of a certain set of parameters, such as the choice of a particular variant of blocking or windowing, the selection of appropriate window size for different datasets etc.MethodsIn this paper, we have proposed a framework that employs blocking and windowing techniques in succession, such that figuring out the parameters is not required. We have also evaluated the impact of different configurations on dirty and massively dirty datasets. To evaluate the proposed framework, experiments are performed using Febrl (Freely Extensible Biomedical Record Linkage).ResultsThe proposed framework is comprehensively evaluated using a variety of quality and complexity parameters such as reduction ratio, precision, recall etc. It is observed that the proposed framework significantly reduces the number of record comparisons.ConclusionsThe selection of the linkage key is a critical performance factor for record linkage.Electronic supplementary materialThe online version of this article (doi:10.1186/s12911-016-0280-9) contains supplementary material, which is available to authorized users.

Planning meets Data Cleansing

One of the motivations for research in data quality is to automatically identify cleansing activities, namely a sequence of actions able to cleanse a dirty dataset, which today are often developed manually by domain-experts. Here we explore the idea that AI Planning can contribute to identify data inconsistencies and automatically fix them. To this end, we formalise the concept of cost-optimal Universal Cleanser — a collection of cleansing actions for each data inconsistency — as a planning problem. We present then a motivating government application in which it has be used.

Towards data cleansing via planning

Nowadays Information Systems generate a lot of data for supporting the activities of firms, organisations, and state agencies. While on the one hand such data are primarily collected for realising domain-specific services (e.g., state agencies use data for managing healthcare and retirement contributions) on the other hand domain analysts aim at using such data for studying the dynamics of subjects' behaviours or phenomena over time. Thus, the quality of data plays a key role in ensuring the effectiveness of the overall knowledge discovery process. In such a context, most of the research on data quality is aimed at automatically identifying cleansing activities, namely a sequence of actions able to cleanse a dirty dataset, which are often developed and coded manually requiring a relevant effort for domain-experts. This work is concerned with using AI Planning for both modelling data quality requirements and automatically identifying

Supervised control of the carrot fly in the Netherlands; insights from a dirty dataset1

‘De Groene Vlieg’ (the Green Fly) is an independent company offering various services concerning integrated pest management in open field agriculture in the Netherlands. One such service is the supervised control of carrot fly (Psila rosae). From 1993 to 2008 the company has monitored the carrot fly population on a total of nearly 65 000 hectares resulting in an extensive dataset. The data was not collected according to proper experimental methods, hence it is titled dirty, but the size of the dataset does allow valuable insights to be gained. De Groene Vlieg monitors the carrot fly population on each field separately. Sticky traps are collected weekly and flies are counted in the laboratory. Growers receive messages containing advice regarding when to apply insecticides and they are responsible for doing so. Overall, over a third of the fields require no spraying at all and for 75% of the fields growers are advised to spray three times or less. One insight from the dataset is that small carrots require more insecticide treatments than large carrots (‘winter carrots’). Reasons for this difference are probably both regional and cultural. Additionally, focusing on a small and uniform area reveals that carrot fly population dynamics are highly variable between years. Another insight is that sowing date has a large effect on the number of insecticide sprayings that are required. Fields sown before the first flight have more flies in the second flight than fields sown later. Comparison of biological and conventional carrot fields shows that carrot fly populations are similar for the first flight. However, during the second flight the number of flies is much higher on organic fields than on conventional ones. Finally, insecticide seed treatment with chlorfenvinphos to control the reproduction of the first flight appears slightly more efficient than supervised control. However, supervised control seems as efficient as seed treatment with clothianidin and beta cyfluthrin.

An extreme-value analysis of wind speeds at five Canadian locations

What strikes one most about the data set of this case study is its sheer size; there are approximately 1.7 x 106 observations each of wind speed and wind direction, and 0.6 x 106 observations of surface pressure. The last two were included on the speculation that they might be useful in understanding the behaviour of the observed winds. Indeed, strong spatial gradients of surface pressure are strongly linked to high wind speeds through the geostrophic equation (see Holton 1972), and strong spatial gradients at a particular time are also generally associated with strong temporal gradients of surface pressure. Thus, on the face of it, the surface pressure record should contain useful information about extreme winds. However, because of the size of the data set and the limited amount of time available, this connection has not been pursued. In order to make the data-handling task a manageable one, only the daily maxima of the hourly observations were studied. This decision reduces the total number of wind-speed observations to approximately 69,000. The second thing which impresses one is the fact that this is a very dirty data set, as are almost all long geophysical time series. The observations, one-minute mean wind speeds recorded hourly, are taken subjectively either from a chart recorder or by observing the movements of a dial over a one-minute period. Judging from the instructions which meteorological technicians are given regarding the method for taking these observations, the observations might be better described as subjectively determined one-minute median wind speeds. Observers are instructed to ignore short gusts and lulls. The observations were taken with a variety of instruments over the years. The height and/or location of the measuring instruments has been changed at least once at every station, and the information which is available about the history of these changes is rather scant. The local topography, which has a profound effect on the characteristics of air flow past a given point, has changed over time with encroaching urbanization and the gradual development of the airports at which the instruments are located. Additional complications which are present in most geophysical time series include the effect of the seasonal cycle on the characteristics of the wind speed and the fact that the atmosphere exhibits variability on long time scales which is not seen in textbook weakly stationary stochastic processes.