Missing Value Imputation Methods Research Articles

Imputation Analysis for Time Series Air Quality (PM10) Data Set: A Comparison of Several Methods

Good quality data is important to guarantee for the best quality results of research analysis. However, the quality of the data often being impacted by the existence of missing values that bring bad implication on the accuracy of analysis and subsequently lead to biased results. In air quality data set, missing values problem often caused by various reasons, for example machine malfunction and errors, computer system crashes, human error and insufficient sampling used. In the case for time series modelling, complete series of data is very important to enable for the model construction. This paper aims to highlight a systematic statistical procedure and analysis on how to investigate the performance of several missing values imputation methods to solve for the problem of missing value existence when data are time series. The knowledge could help researchers to implement a comprehensive procedure in deciding a type of imputation method that suits with their data. A case study was conducted using real data set from Shah Alam air quality monitoring station. The results have shown that the missing data at the monitoring station is completely at random (MCAR). Among six imputation methods compared and based on the performance of indicators such as RMSE, MAE, AI and R2 it is shown that imputation using Kalman Filter using ARIMA model is the best appropriate method for the data set.

A Novel Fuzzy Rough Clustering Parameter-based missing value imputation

For a long time, missing values are the biggest challenging problem in data mining, machine learning and statistical analysis. In the current scenario, various methods exist to handle the missing values as it’s an important task to discover meaningful information. However, the most frequently used method to handle the missing values in a large dataset is discarding the instances with missing values. In such situation, deletion of instances with missing values causes loss of crucial information, which affects the performance of algorithms. Hence, an intelligent method needs to handle the missing values. In the recent past, the fuzzy and rough set has been widely employed in many applications. In this research work, a Novel Fuzzy C-Means Rough Parameter-based missing value imputation method is proposed with the hybridization of the fuzzy and rough set to handle missing values. The proposed algorithm is capable of handling the situation of uncertainty and vagueness in datasets through rough and fuzzy sets while maintaining vital information. The experimentation has been carried out on three benchmark datasets such as the Dukes’ B colon cancer dataset, the Mice Protein Expression and Yeast datasets to asses the efficacy of the proposed method. It is observed that the proposed method produces improved results than Fuzzy C-Means Centroid-based missing value imputation and Fuzzy C-Means Parameter-based missing value imputation method.

An industrial missing values processing method based on generating model

The issue of missing values (MVs) has been found widely in real-world datasets and obstructed the use of many statistical or machine learning algorithms for data analytics due to their incompetence in processing incomplete datasets. Most of the current MVs filling methods are applied to the datasets with certain specific types or low missing rate. This paper proposes a method of missing values processing based on the combination of denoising autoencoder (DAE) and generative adversarial networks (GAN), aiming at the missing completely at random (MCAR) datasets with high missing rate and noise interference in industrial scenes. We execute the training process on a discrete dataset with missing values, in order to ensure the generated dataset is completely similar to the feature distribution of the original dataset. We conduct our experiments for different dimensional datasets to prove the feasibility and efficiency of this method, including three public authority datasets and an industrial production monitoring dataset. The results compared with traditional missing values imputation methods have shown when the missing rate is higher than 30%, our method performs better in robustness and accuracy.

The efficiency of multiple imputation and maximum likelihood methods for estimating missing values

Objectives: This study investigated the efficiency of Multiple Imputation (MI) and Maximum Likelihood (ML) methods for estimating missing values. The study was set to use the findings to make recommendations for future studies about the impact of missing data imputation on the accuracy of Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). Methods: The completedset (with no missing values) used in this study was collected in 2010/11 through the Income and Expenditure Survey (IES) and had 25328 observations. Missing data were generated by randomly deleting 10%, 20%, 30%, 40% and 50% of the values from the complete dataset. The missing values in each of the five datasets were imputed using MI and ML methods. Subsequently, absolute error values of AIC and BIC from multiple regression analysis were computed for each dataset. The study then compared the absolute errors for each missing value imputation method. Findings: The findings of the study revealed that AIC and BIC are more accurate when missing values are estimated by the Full Information Maximum Likelihood (FIML) of the ML algorithm, provided 10% of the data are missing. For all datasets, AIC and BIC were least accurate when missing values were imputed by Expectation Maximisation (EM) of the ML algorithm. The findings also showed that AIC and BIC are more accurate when the rate of MISSINGNESS gets large provided missing values were estimated using either the Fully Conditional Specification (FCS) or Markov Chain Monte Carlo (MCMC), MI algorithms. Application: When the rate of MISSINGNESS is small (at most 10%), FIML should be used to handle missing data if AIC and BIC are going to be used. Also both FCS and MCMC should be considered over EM algorithms when the rate of MISSINGNESS is high (at least 40% missing). Keywords: Maximum Likelihood Imputation, Multiple Imputation, AIC, BIC

A class center based approach for missing value imputation

Missing value imputation (MVI) is the major solution method for dealing with incomplete dataset problems in which the missing attribute values are replaced from a chosen set of observed data using some statistical methods, such as mean/mode, machine learning, or support vector machine methods. Although machine learning MVI approaches may produce reasonably good imputation results, they usually require larger imputation times than statistical approaches. In this paper, a Class Center based Missing Value Imputation (CCMVI) approach is introduced for producing effective imputation results more efficiently. It is based on measuring the class center of each class and then the distances between it and the other observed data are used to define a threshold for the later imputation. The experimental results based on numerical, categorical, and mixed data types of datasets show that the proposed CCMVI approach outperforms the other MVI approaches for both numerical and mixed datasets. In addition, it requires much less imputation time than the machine learning MVI methods.

GSimp: A Gibbs sampler based left-censored missing value imputation approach for metabolomics studies.

Left-censored missing values commonly exist in targeted metabolomics datasets and can be considered as missing not at random (MNAR). Improper data processing procedures for missing values will cause adverse impacts on subsequent statistical analyses. However, few imputation methods have been developed and applied to the situation of MNAR in the field of metabolomics. Thus, a practical left-censored missing value imputation method is urgently needed. We developed an iterative Gibbs sampler based left-censored missing value imputation approach (GSimp). We compared GSimp with other three imputation methods on two real-world targeted metabolomics datasets and one simulation dataset using our imputation evaluation pipeline. The results show that GSimp outperforms other imputation methods in terms of imputation accuracy, observation distribution, univariate and multivariate analyses, and statistical sensitivity. Additionally, a parallel version of GSimp was developed for dealing with large scale metabolomics datasets. The R code for GSimp, evaluation pipeline, tutorial, real-world and simulated targeted metabolomics datasets are available at: https://github.com/WandeRum/GSimp.

In-depth method assessments of differentially expressed protein detection for shotgun proteomics data with missing values

Considering as one of the major goals in quantitative proteomics, detection of the differentially expressed proteins (DEPs) plays an important role in biomarker selection and clinical diagnostics. There have been plenty of algorithms and tools focusing on DEP detection in proteomics research. However, due to the different application scopes of these methods, and various kinds of experiment designs, it is not very apparent about the best choice for large-scale proteomics data analyses. Moreover, given the fact that proteomics data usually contain high percentage of missing values (MVs), but few replicates, a systematic evaluation of the DEP detection methods combined with the MV imputation methods is essential and urgent. Here, we analyzed a total of four representative imputation methods and five DEP methods on different experimental and simulated datasets. The results showed that (i) MV imputation could not always improve the performances of DEP detection methods and the imputation effects differed in the missing value percentages; (ii) the DEP detection methods had different statistical powers affected by the percentage of MVs. Two statistical methods (i.e. the empirical Bayesian random censoring threshold model, and the significance analysis of microarray) performed better than the other evaluated methods in terms of accuracy and sensitivity.

An Overview and Evaluation of Recent Machine Learning Imputation Methods Using Cardiac Imaging Data.

Many clinical research datasets have a large percentage of missing values that directly impacts their usefulness in yielding high accuracy classifiers when used for training in supervised machine learning. While missing value imputation methods have been shown to work well with smaller percentages of missing values, their ability to impute sparse clinical research data can be problem specific. We previously attempted to learn quantitative guidelines for ordering cardiac magnetic resonance imaging during the evaluation for pediatric cardiomyopathy, but missing data significantly reduced our usable sample size. In this work, we sought to determine if increasing the usable sample size through imputation would allow us to learn better guidelines. We first review several machine learning methods for estimating missing data. Then, we apply four popular methods (mean imputation, decision tree, k-nearest neighbors, and self-organizing maps) to a clinical research dataset of pediatric patients undergoing evaluation for cardiomyopathy. Using Bayesian Rule Learning (BRL) to learn ruleset models, we compared the performance of imputation-augmented models versus unaugmented models. We found that all four imputation-augmented models performed similarly to unaugmented models. While imputation did not improve performance, it did provide evidence for the robustness of our learned models.

Evaluation of missing value imputation methods for wireless soil datasets

Soil data are very important for hydrologists to model and predict the evolution of water---soil environments. In present, the soil data are often collected by unattended wireless sensing system and then inevitably involves continuous missing values due to the unreliability of system, which is different from the manually collected datasets with the data losses being sparsely distributed . This paper investigates seven typical methods that are used to infill soil missing data, and in particular we also attempt to employ the extreme learning machine in missing-data infilling. This work is aimed at answering such a question: Whether or not existing methods suit for wireless sensory soil dataset with continuous missing values, and how well they perform. With a real-world soil dataset involving complete samples as the benchmark, we evaluate and compare these methods , and analyze the possible reasons behind. This study provides insights for designing new methods that can effectively deal with the missing values in wireless sensory soil dataset.

Missing data imputation using fuzzy-rough methods

Missing values exist in many generated datasets in science. Therefore, utilizing missing data imputation methods is a common and important practice. These methods are a kind of treatment for uncertainty and vagueness existing in datasets. On the other hand, methods based on fuzzy-rough sets provide excellent tools for dealing with uncertainty, possessing highly desirable properties such as robustness and noise tolerance. Furthermore, they can find minimal representations of data and do not need potentially erroneous user inputs. As a result, utilizing fuzzy-rough sets for imputation should be an effective approach. In this paper, we propose three missing value imputation methods based on fuzzy-rough sets and its recent extensions; namely, implicator/t-norm based fuzzy-rough sets, vaguely quantified rough sets and also ordered weighted average based rough sets. These methods are compared against 11 state-of-the-art imputation methods implemented in the KEEL data mining software on 27 benchmark datasets. The results show, via non-parametric statistical analysis, that the proposed methods exhibit excellent performance in general.

On mining incomplete medical datasets: Ordering imputation and classification.

To collect medical datasets, it is usually the case that a number of data samples contain some missing values. Performing the data mining task over the incomplete datasets is a difficult problem. In general, missing value imputation can be approached, which aims at providing estimations for missing values by reasoning from the observed data. Consequently, the effectiveness of missing value imputation is heavily dependent on the observed data (or complete data) in the incomplete datasets. In this paper, the research objective is to perform instance selection to filter out some noisy data (or outliers) from a given (complete) dataset to see its effect on the final imputation result. Specifically, four different processes of combining instance selection and missing value imputation are proposed and compared in terms of data classification. Experiments are conducted based on 11 medical related datasets containing categorical, numerical, and mixed attribute types of data. In addition, missing values for each dataset are introduced into all attributes (the missing data rates are 10%, 20%, 30%, 40%, and 50%). For instance selection and missing value imputation, the DROP3 and k-nearest neighbor imputation methods are employed. On the other hand, the support vector machine (SVM) classifier is used to assess the final classification accuracy of the four different processes. The experimental results show that the second process by performing instance selection first and imputation second allows the SVM classifiers to outperform the other processes. For incomplete medical datasets containing some missing values, it is necessary to perform missing value imputation. In this paper, we demonstrate that instance selection can be used to filter out some noisy data or outliers before the imputation process. In other words, the observed data for missing value imputation may contain some noisy information, which can degrade the quality of the imputation result as well as the classification performance.

Missing value imputation method for disaster decision-making using K nearest neighbor

Due to destructiveness of natural disasters, restriction of disaster scenarios and some human causes, missing data usually occur in disaster decision-making problems. In order to estimate missing values of alternatives, this paper focuses on imputing heterogeneous attribute values of disaster based on an improved K nearest neighbor imputation (KNNI) method. Firstly, some definitions of trapezoidal fuzzy numbers (TFNs) are introduced and three types of attributes (i.e. linguistic term sets, intervals and real numbers) are converted to TFNs. Then the correlated degree model is utilized to extract related attributes to form instances that will be used in K nearest neighbor algorithm, and a novel KNNI method merging with correlated degree model is presented. Finally, an illustrative example is given to verify the proposed method and to demonstrate its feasibility and effectiveness.

Applications of temporal kernel canonical correlation analysis in adherence studies.

Adherence to medication is often measured as a continuous outcome but analyzed as a dichotomous outcome due to lack of appropriate tools. In this paper, we illustrate the use of the temporal kernel canonical correlation analysis (tkCCA) as a method to analyze adherence measurements and symptom levels on a continuous scale. The tkCCA is a novel method developed for studying the relationship between neural signals and hemodynamic response detected by functional MRI during spontaneous activity. Although the tkCCA is a powerful tool, it has not been utilized outside the application that it was originally developed for. In this paper, we simulate time series of symptoms and adherence levels for patients with a hypothetical brain disorder and show how the tkCCA can be used to understand the relationship between them. We also examine, via simulations, the behavior of the tkCCA under various missing value mechanisms and imputation methods. Finally, we apply the tkCCA to a real data example of psychotic symptoms and adherence levels obtained from a study based on subjects with a first episode of schizophrenia, schizophreniform or schizoaffective disorder.

Impact of missing data imputation methods on gene expression clustering and classification.

BackgroundSeveral missing value imputation methods for gene expression data have been proposed in the literature. In the past few years, researchers have been putting a great deal of effort into presenting systematic evaluations of the different imputation algorithms. Initially, most algorithms were assessed with an emphasis on the accuracy of the imputation, using metrics such as the root mean squared error. However, it has become clear that the success of the estimation of the expression value should be evaluated in more practical terms as well. One can consider, for example, the ability of the method to preserve the significant genes in the dataset, or its discriminative/predictive power for classification/clustering purposes.Results and conclusionsWe performed a broad analysis of the impact of five well-known missing value imputation methods on three clustering and four classification methods, in the context of 12 cancer gene expression datasets. We employed a statistical framework, for the first time in this field, to assess whether different imputation methods improve the performance of the clustering/classification methods. Our results suggest that the imputation methods evaluated have a minor impact on the classification and downstream clustering analyses. Simple methods such as replacing the missing values by mean or the median values performed as well as more complex strategies. The datasets analyzed in this study are available at http://costalab.org/Imputation/.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-015-0494-3) contains supplementary material, which is available to authorized users.

Use of biplot technique for the comparison of the missing value imputation methods

This study was performed to assess the effects of different imputation methods on the performance of a biplot technique. We selected the Fisher's iris data as our reference dataset. Some elements of the Iris data were deleted in different rates under missing at random MAR assumption to generate incomplete datasets which had 3.5%, 7%, 15%, 20% missing value. Datasets with missing values were completed by four imputation methods [mean imputation, regression imputation, expectation maximisation EM algorithm, multiple imputation MI]. The new imputed datasets were analysed by biplot technique and their results were compared with original complete biplot of the data. The results of biplot analysis were similar in all the imputation methods when missing rate is low under MAR assumption. Even when the missing rate was greater than 10%, results of EM and MI methods were similar to real values and graphical representation of original data. For multivariate methods, we also propose filling in the missing value with the arithmetic mean of the imputed estimates which are obtained with multiple imputation. This paper also indicates that the use of biplot technique for the comparison of the missing value imputation methods provides a useful visual tool.

Missing value imputation in high-dimensional phenomic data: imputable or not, and how?

BackgroundIn modern biomedical research of complex diseases, a large number of demographic and clinical variables, herein called phenomic data, are often collected and missing values (MVs) are inevitable in the data collection process. Since many downstream statistical and bioinformatics methods require complete data matrix, imputation is a common and practical solution. In high-throughput experiments such as microarray experiments, continuous intensities are measured and many mature missing value imputation methods have been developed and widely applied. Numerous methods for missing data imputation of microarray data have been developed. Large phenomic data, however, contain continuous, nominal, binary and ordinal data types, which void application of most methods. Though several methods have been developed in the past few years, not a single complete guideline is proposed with respect to phenomic missing data imputation.ResultsIn this paper, we investigated existing imputation methods for phenomic data, proposed a self-training selection (STS) scheme to select the best imputation method and provide a practical guideline for general applications. We introduced a novel concept of “imputability measure” (IM) to identify missing values that are fundamentally inadequate to impute. In addition, we also developed four variations of K-nearest-neighbor (KNN) methods and compared with two existing methods, multivariate imputation by chained equations (MICE) and missForest. The four variations are imputation by variables (KNN-V), by subjects (KNN-S), their weighted hybrid (KNN-H) and an adaptively weighted hybrid (KNN-A). We performed simulations and applied different imputation methods and the STS scheme to three lung disease phenomic datasets to evaluate the methods. An R package “phenomeImpute” is made publicly available.ConclusionsSimulations and applications to real datasets showed that MICE often did not perform well; KNN-A, KNN-H and random forest were among the top performers although no method universally performed the best. Imputation of missing values with low imputability measures increased imputation errors greatly and could potentially deteriorate downstream analyses. The STS scheme was accurate in selecting the optimal method by evaluating methods in a second layer of missingness simulation. All source files for the simulation and the real data analyses are available on the author’s publication website.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-014-0346-6) contains supplementary material, which is available to authorized users.

A functional data approach to missing value imputation and outlier detection for traffic flow data

Missing values and outliers are frequently encountered in traffic monitoring data. We approach these problems by sampling the daily traffic flow rate trajectories from random functions and taking advantage of the data features using functional data analysis. We propose to impute missing values by using the conditional expectation approach to functional principal component analysis (FPCA). Our simulation study shows that the FPCA approach performs better than two commonly discussed methods in the literature, the probabilistic principal component analysis (PCA) and the Bayesian PCA, which have been shown to perform better than many conventional approaches. Based on the FPCA approach, the functional principal component scores can be applied to the functional bagplot and functional highest density region boxplot, which makes outlier detection possible for incomplete functional data. Our numerical results indicate that these two outlier detection approaches coupled with the proposed missing value imputation method can perform reasonably well. Although motivated by traffic flow data application, the proposed functional data methods for missing value imputation and outlier detection can be used in many applications with longitudinally recorded functional data.

A Review on Missing Value Imputation Algorithms for Microarray Gene Expression Data

Many bioinformatics analytical tools, especially for cancer classification and prediction, require complete sets of data matrix. Having missing values in gene expression studies significantly influences the interpretation of final data. However, to most analysts' dismay, this has become a common problem and thus, relevant missing value imputation algorithms have to be developed and/or refined to address this matter. This paper intends to present a review of preferred and available missing value imputation methods for the analysis and imputation of missing values in gene expression data. Focus is placed on the abilities of algorithms in performing local or global data correlation to estimate the missing values. Approaches of the algorithms mentioned have been categorized into global approach, local approach, hybrid approach, and knowledge assisted approach. The methods presented are accompanied with suitable performance evaluation. The aim of this review is to highlight possible improvements on existing research techniques, rather than recommending new algorithms with the same functional aim.

Prediction of regulatory gene pairs using dynamic time warping and gene ontology.

Selecting informative genes is the most important task for data analysis on microarray gene expression data. In this work, we aim at identifying regulatory gene pairs from microarray gene expression data. However, microarray data often contain multiple missing expression values. Missing value imputation is thus needed before further processing for regulatory gene pairs becomes possible. We develop a novel approach to first impute missing values in microarray time series data by combining k-Nearest Neighbour (KNN), Dynamic Time Warping (DTW) and Gene Ontology (GO). After missing values are imputed, we then perform gene regulation prediction based on our proposed DTW-GO distance measurement of gene pairs. Experimental results show that our approach is more accurate when compared with existing missing value imputation methods on real microarray data sets. Furthermore, our approach can also discover more regulatory gene pairs that are known in the literature than other methods.

Performance Improvement of Fuzzy and Neuro Fuzzy Systems: Prediction of Learning Disabilities in School-age Children

Learning Disability (LD) is a classification including several disorders in which a child has difficulty in learning in a typical manner, usually caused by an unknown factor or factors. LD affects about 15% of children enrolled in schools. The prediction of learning disability is a complicated task since the identification of LD from diverse features or signs is a complicated problem. There is no cure for learning disabilities and they are life-long. The problems of children with specific learning disabilities have been a cause of concern to parents and teachers for some time. The aim of this paper is to develop a new algorithm for imputing missing values and to determine the significance of the missing value imputation method and dimensionality reduction method in the performance of fuzzy and neuro fuzzy classifiers with specific emphasis on prediction of learning disabilities in school age children. In the basic assessment method for prediction of LD, checklists are generally used and the data cases thus collected fully depends on the mood of children and may have also contain redundant as well as missing values. Therefore, in this study, we are proposing a new algorithm, viz. the correlation based new algorithm for imputing the missing values and Principal Component Analysis (PCA) for reducing the irrelevant attributes. After the study, it is found that, the preprocessing methods applied by us improves the quality of data and thereby increases the accuracy of the classifiers. The system is implemented in Math works Software Mat Lab 7.10. The results obtained from this study have illustrated that the developed missing value imputation method is very good contribution in prediction system and is capable of improving the performance of a classifier.