Data Preparation Tool Research Articles

Cleenex: Support for User Involvement during an Iterative Data Cleaning Process

The existence of large amounts of data increases the probability of occurring data quality problems. A data cleaning process that corrects these problems is usually an iterative process, because it may need to be re-executed and refined to produce high-quality data. Moreover, due to the specificity of some data quality problems and the limitation of data cleaning programs to cover all problems, often a user has to be involved during the program executions by manually repairing data. However, there is no data cleaning framework that appropriately supports this involvement in such an iterative process, a form of human-in-the-loop, to clean structured data. Moreover, data preparation tools that somehow involve the user in data cleaning processes have not been evaluated with real users to assess their effort. Therefore, we propose Cleenex, a data cleaning framework with support for user involvement during an iterative data cleaning process, and conduct two data cleaning experimental evaluations: an assessment of the Cleenex components that support the user when manually repairing data with a simulated user; and a comparison, in terms of user involvement, of data preparation tools with real users. Results show that Cleenex components reduce the user effort when manually cleaning data during a data cleaning process, for example, the number of tuples visualized is reduced in 99%. Moreover, when performing data cleaning tasks with Cleenex, real users need less time/effort (e.g., half the clicks) and, based on questionnaires, prefer it to the other tools used for comparison, OpenRefine and Pentaho Data Integration.

DATA MINING PREPARATION: PROCESS, TECHNIQUES AND MAJOR ISSUES IN DATA ANALYSIS

Data preparation is an iterative-agile process for exploring, combining, cleaning and transforming raw data into curated datasets for self-service data integration, data science, data discovery, and BI/analytics. To perform data preparation, data preparation tools are used by analysts, citizen data scientists and data scientists for self- service. The tools are also used by citizen integrators and data engineers for data enablement to reduce the time and complexity of interactively accessing, cataloging, harmonizing, transforming and modeling data for analytics in an agile manner with metadata and lineage support. These tools can provide data access for use in mostly analytical tasks that include storage, logical and physical data modeling, and data manipulation for data visualization, data integration and analytics. Some tools support machine-learning algorithms that can recommend or even automate actions to augment and accelerate data preparation. Keywords: Data Mining, Data preparation, Data Analysis, Big Data Analytics

Data Preparation

Raw data are often messy: they follow different encodings, records are not well structured, values do not adhere to patterns, etc. Such data are in general not fit to be ingested by downstream applications, such as data analytics tools, or even by data management systems. The act of obtaining information from raw data relies on some data preparation process. Data preparation is integral to advanced data analysis and data management, not only for data science but for any data-driven applications. Existing data preparation tools are operational and useful, but there is still room for improvement and optimization. With increasing data volume and its messy nature, the demand for prepared data increases day by day. To cater to this demand, companies and researchers are developing techniques and tools for data preparation. To better understand the available data preparation systems, we have conducted a survey to investigate (1) prominent data preparation tools, (2) distinctive tool features, (3) the need for preliminary data processing even for these tools and, (4) features and abilities that are still lacking. We conclude with an argument in support of automatic and intelligent data preparation beyond traditional and simplistic techniques.

Effect of heat accumulation on the single track formation during laser metal deposition and development of a framework for analyzing new process strategies

Laser metal deposition (LMD) is getting more attention as a 3D printing technique recently. Compared to the cladding application, the process of building 3D structures is more challenging to control when it comes to complex structures. Changing heat transfer mechanisms over the building height and at critical part geometries can lead to instabilities during the process. This paper studies the effect of the surface temperature on single track geometry formation during the LMD process. The geometrical characteristic of the track formation is a key factor that needs to be controlled and monitored in any deposition process for a stable build job. To quantify the effect of the surface temperature on the track geometry, a stepwise preheated substrate from 25 to 500 °C is used, and the track height, width, and degree of dilution are evaluated. This experiment mimics the varying temperature of the prebuilt layers that occur during additive manufacturing of freeform structures with many layers. Understanding the influence of the surface temperature over the layer geometry gives an idea of build geometry variation due to the accumulated heat input along the build direction. This helps in defining process strategies and process parameters for the laser metal deposition of components with increased accuracy and reproducibility. Advanced process strategies that are essential for a successful build job need to be incorporated into a data preparation tool for robot code generation. Commercial software packages are available for regular additive path planning using industrial robots. However, when several process strategies and process parameters need to be adapted and varied along the build direction, one is pushed to the limits of the software capabilities. Therefore, an automated data preprocessing toolbox based on MATLAB has been developed for the implementation of process strategies through user-definable rules and algorithms. This eliminates manual robot code preparation and correction. Implementation and the level of automation of this process preparation toolbox have also been discussed in this paper.

Discovering and merging related analytic datasets

The production of analytic datasets is a significant big data trend and has gone well beyond the scope of traditional IT-governed dataset development. Analytic datasets are now created by data scientists and data analysts using big data frameworks and agile data preparation tools. However, despite the profusion of available datasets, it remains quite difficult for a data analyst to start from a dataset at hand and customize it with additional attributes coming from other existing datasets. This article describes a model and algorithms that exploit automatically extracted and user-defined semantic relationships for extending analytic datasets with new atomic or aggregated attribute values. Our framework is implemented as a REST service in SAP HANA and includes a careful theoretical analysis and practical solutions for several complex data quality issues.

A New Method to Detect Outliers in High-frequency Time Series

The objective of this research is to develop a fast, simple method for detecting and replacing extreme spikes in high-frequency time series data. The method primarily consists  of a nonparametric procedure that pursues a balance between fidelity to observed data and smoothness. Furthermore, through examination of the absolute difference between original and smoothed values, the technique is also able to detect and, where necessary, replace outliers with less extreme data.
 Unlike other filtering procedures found in the literature, our method does not require a model to be specified for the data. Additionally, the filter makes only a single pass through the time series. Experiments  show that the new method can be validly used as a data preparation tool to ensure that time series modeling is supported by clean data, particularly in a complex context such as one with high-frequency data.

START: A data preparation tool for crop simulation models using web-based soil databases

Soil profile data that characterize the physical and chemical properties of a soil are among the required set of inputs for ecological, crop and other dynamic simulation models. A web-based soil information system often provides the site-specific soil data of which formats are not readily compatible to crop models. The Soil daTA Retrieval Tool (START) was developed to automate a series of procedures for preparation of soil input data that includes the retrieval of soil profile data from the information system, reorganization of data, estimation of soil parameters, and creation of input files for simulation models. In a case study, the START was implemented to support the SoilGrids database operated by the International Soil Reference and Information Center. It took about 0.33% of time for the START to create soil input files compared with manual preparation. These results suggest that the START could provide an efficient approach for preparation of soil input files especially for sites where little soil information is available.

Critical Care Health Informatics Collaborative (CCHIC): Data, tools and methods for reproducible research: A multi-centre UK intensive care database

ObjectiveTo build and curate a linkable multi-centre database of high resolution longitudinal electronic health records (EHR) from adult Intensive Care Units (ICU).To develop a set of open-source tools to make these data ‘research ready’ while protecting patient’s privacy with a particular focus on anonymisation. Materials and methodsWe developed a scalable EHR processing pipeline for extracting, linking, normalising and curating and anonymising EHR data. Patient and public involvement was sought from the outset, and approval to hold these data was granted by the NHS Health Research Authority’s Confidentiality Advisory Group (CAG). The data are held in a certified Data Safe Haven. We followed sustainable software development principles throughout, and defined and populated a common data model that links to other clinical areas. ResultsLongitudinal EHR data were loaded into the CCHIC database from eleven adult ICUs at 5 UK teaching hospitals. From January 2014 to January 2017, this amounted to 21,930 and admissions (18,074 unique patients). Typical admissions have 70 data-items pertaining to admission and discharge, and a median of 1030 (IQR 481–2335) time-varying measures. Training datasets were made available through virtual machine images emulating the data processing environment. An open source R package, cleanEHR, was developed and released that transforms the data into a square table readily analysable by most statistical packages. A simple language agnostic configuration file will allow the user to select and clean variables, and impute missing data. An audit trail makes clear the provenance of the data at all times. DiscussionMaking health care data available for research is problematic. CCHIC is a unique multi-centre longitudinal and linkable resource that prioritises patient privacy through the highest standards of data security, but also provides tools to clean, organise, and anonymise the data. We believe the development of such tools are essential if we are to meet the twin requirements of respecting patient privacy and working for patient benefit. ConclusionThe CCHIC database is now in use by health care researchers from academia and industry. The ‘research ready' suite of data preparation tools have facilitated access, and linkage to national databases of secondary care is underway.

Utilization of AERONET polarimetric measurements for improving retrieval of aerosol microphysics: GSFC, Beijing and Dakar data analysis

The study presents the efforts on including the polarimetric data to the routine inversion of the radiometric ground-based measurements for characterization of the atmospheric aerosols and analysis of the obtained advantages in retrieval results. First, to operationally process the large amount of polarimetric data the data preparation tool was developed. The AERONET inversion code adapted for inversion of both intensity and polarization measurements was used for processing. Second, in order to estimate the effect from utilization of polarimetric information on aerosol retrieval results, both synthetic data and the real measurements were processed using developed routine and analyzed. The sensitivity study has been carried out using simulated data based on three main aerosol models: desert dust, urban industrial and urban clean aerosols. The test investigated the effects of utilization of polarization data in the presence of random noise, bias in measurements of optical thickness and angular pointing shift. The results demonstrate the advantage of polarization data utilization in the cases of aerosols with pronounced concentration of fine particles. Further, the extended set of AERONET observations was processed. The data for three sites have been used: GSFC, USA (clean urban aerosol dominated by fine particles), Beijing, China (polluted industrial aerosol characterized by pronounced mixture of both fine and coarse modes) and Dakar, Senegal (desert dust dominated by coarse particles). The results revealed considerable advantage of polarimetric data applying for characterizing fine mode dominated aerosols including industrial pollution (Beijing). The use of polarization corrects particle size distribution by decreasing overestimated fine mode and increasing the coarse mode. It also increases underestimated real part of the refractive index and improves the retrieval of the fraction of spherical particles due to high sensitivity of polarization to particle shape. Overall, the study demonstrates a substantial value of polarimetric data for improving aerosol characterization.

A Bayesian imputation method for a clustering genetic algorithm

Missing values are a critical problem in data mining applications. The substitution of these values, also called imputation, can be performed by several methods. This work describes the application of an optimized version of the Bayesian Algorithm K2 as an imputation tool for a clustering genetic algorithm. The resulting hybrid system is assessed by means of simulations in five benchmark datasets. The obtained results indicate that the proposed imputation method is a suitable data preparation tool for the employed clustering genetic algorithm.

Using Relational Database Software To Prepare An Archival Dataset For Importation Into A Statistical Analysis Package

Researchers frequently find that archival data that they need to test their theoretical models must undergo a variety of preparatory steps before the data are ready to be loaded into their favorite statistical analysis package. Although some powerful software tools and techniques to handle such data preparation do exist and are relatively easy to learn, many researchers remain unaware of them. The purpose of this paper is to introduce some of these data preparation tools and techniques by use of a hands-on demonstration. The demonstration begins with data embedded in a Web page and shows how a spreadsheet, a word processor, and the SQL language from a relational database can be used to make the extracted “ready for prime-time.” The original Web page data for the demonstration can be downloaded from the authors’ web page; the software packages used to prepare the data are MS Excel, MS Word, and MS Access. Although SPSS is used for the statistical analysis described in the final phase of the process, readers do not have to have this software to benefit from the demonstration. Any statistical analysis package could be used.

PreLALS Workbench: Visual Data Manipulation Workbench for WinLALS on Windows PCs

PreLALS workbench is designed to aid users in the task of visualizing chemical structures using three-dimensional models, editing and the data manipulation for WinLALS program that refines the internal coordinates based on linked-atom least-squares (LALS) method for helical polymers. It combines the views of model building (MB), analyses, and data preparation tools with easy-to-use graphical user interfaces. We describe the outline of the WinLALS program with mathematical expressions and explain the several functions of the PreLALS workbench using each menu of the data preparation and example dialogs. The PreLALS workbench runs on most IBM compatible Windows PCs having a minimum of Pentium and the operating system must be either Windows 2000, 98, Me or XP.