Data Imputation Approaches Research Articles

Data Imputation in Merged Isobaric Labeling-Based Relative Quantification Datasets.

The data-dependent acquisition in mass spectrometry-based proteomics combined with quantitative analysis using isobaric labeling (iTRAQ and TMT) inevitably introduces missing values in proteomic experiments where a number of LC-runs are combined, especially in the growing field of shotgun clinical proteomics, where the protein profiles from the proteomics analysis of several hundred patient samples are compared and correlated to clinical traits such as a specific disease or disease treatment in order to link specific outcomes to one or more proteins. In the context of clinical research it is evident that missing values in such datasets reduce the power of the downstream statistical analysis therefore may hampers the linking of the expression of disease traits to the expression of specific proteins that may be useful for prognostic, diagnostic, or predictive purposes. In our study, we tested three data imputation approaches initially developed for microarray data for the imputation of missing values in datasets that are generated by several runs of shotgun proteomic experiments and where the data were relative protein abundances based on isobaric tags (iTRAQ and TMT). Our conclusion is that imputation methods based on k Nearest Neighbors successfully impute missing values in datasets with up to 50% missing values.

Medical card data imputation and patient psychological and behavioral profile construction

Missing data is a typical problem for many hands-on tasks and researches, which has required human intervention and contributed to an increase in errors during algorithms application that demand for a large number of metrics. Solving this particular problem is essential for medicine and healthcare, because it allows more easily diagnosing certain types of diseases, improving medical service quality, etc. The main approach for medical data imputation is to automate this process at all stages, beginning from finding the NA (Not Available) or missing Data, to the completion of the analysis and insertion of lost information entity. The proposed methods of mathematical computing and modeling, statistical functions, data flow diagrams during the imputation, and the use of computer programming tools should be implemented in the medical field to improve and address the missing data issue. The evaluation of key characteristics (algorithm’s error, number of imputed data, datasets dimensionality) helped to determine the factors for obtaining the most accurate result with the help of various algorithms and functions. The study is useful for the medical industry in general, since it will eliminate the missing data values in patient medical records by applying statistical methods and artificial intelligence, which will significantly shorten the automation of large datasets processing and facilitate their descriptive and exploratory analysis during further data discovery to identify certain patterns and features.

Innovative method for traffic data imputation based on convolutional neural network

The quality of traffic data is crucial for modern transportation planning and operations. However, data could be missing for various reasons. Hence, the data imputation approaches which aim at predicting/replacing the missing data or bad data have been considered very important. The traditional traffic data imputation approaches mainly focus on using different probability models or regression methods to impute data, and they only take limited temporal or spatial information as inputs. Thus, they are not very accurate especially for data with a high missing ratio. To overcome the weaknesses of previous approaches, this study proposes an innovative traffic data imputation method, which first transforms the raw data into spatial-temporal images and then implements a deep-learning method on the images. The key idea of this approach is developing a convolutional neural network (CNN)-based context encoder to reconstruct the complete image from the missing source. To the best of the authors’ knowledge, this is the first time a CNN method has been incorporated for traffic data imputation. Experiments are conducted on three months of data from 256 loop detectors. Through comparison with two state-of-the-art approaches, the results indicate that this new approach increases the imputation accuracy greatly and has a stable error distribution.

Addressing missing data in confounders when estimating propensity scores for continuous exposures

Propensity score models are frequently used to estimate causal effects in observational studies. One unresolved issue in fitting these models is handling missing values in the propensity score model covariates. As these models usually contain a large set of covariates, using only individuals with complete data significantly decreases the sample size and statistical power. Several missing data imputation approaches have been proposed, including multiple imputation (MI), MI with missingness pattern (MIMP), and treatment mean imputation. Generalized boosted modeling (GBM), which is a nonparametric approach to estimate propensity scores, can automatically handle missingness in the covariates. Although the performance of MI, MIMP, and treatment mean imputation have previously been compared for binary treatments, they have not been compared for continuous exposures or with single imputation and GBM. We compared these approaches in estimating the generalized propensity score (GPS) for a continuous exposure in both a simulation study and in empirical data. Using GBM with the incomplete data to estimate the GPS did not perform well in the simulation. Missing values should be imputed before estimating propensity scores using GBM or any other approach for estimating the GPS.

An Experimental Evaluation of Fault Diagnosis from Imbalanced and Incomplete Data for Smart Semiconductor Manufacturing

The SECOM dataset contains information about a semiconductor production line, entailing the products that failed the in-house test line and their attributes. This dataset, similar to most semiconductor manufacturing data, contains missing values, imbalanced classes, and noisy features. In this work, the challenges of this dataset are met and many different approaches for classification are evaluated to perform fault diagnosis. We present an experimental evaluation that examines 288 combinations of different approaches involving data pruning, data imputation, feature selection, and classification methods, to find the suitable approaches for this task. Furthermore, a novel data imputation approach, namely “In-painting KNN-Imputation” is introduced and is shown to outperform the common data imputation technique. The results show the capability of each classifier, feature selection method, data generation method, and data imputation technique, with a full analysis of their respective parameter optimizations.

SigEMD: A powerful method for differential gene expression analysis in single-cell RNA sequencing data.

Differential gene expression analysis is one of the significant efforts in single cell RNA sequencing (scRNAseq) analysis to discover the specific changes in expression levels of individual cell types. Since scRNAseq exhibits multimodality, large amounts of zero counts, and sparsity, it is different from the traditional bulk RNA sequencing (RNAseq) data. The new challenges of scRNAseq data promote the development of new methods for identifying differentially expressed (DE) genes. In this study, we proposed a new method, SigEMD, that combines a data imputation approach, a logistic regression model and a nonparametric method based on the Earth Mover's Distance, to precisely and efficiently identify DE genes in scRNAseq data. The regression model and data imputation are used to reduce the impact of large amounts of zero counts, and the nonparametric method is used to improve the sensitivity of detecting DE genes from multimodal scRNAseq data. By additionally employing gene interaction network information to adjust the final states of DE genes, we further reduce the false positives of calling DE genes. We used simulated datasets and real datasets to evaluate the detection accuracy of the proposed method and to compare its performance with those of other differential expression analysis methods. Results indicate that the proposed method has an overall powerful performance in terms of precision in detection, sensitivity, and specificity.

A Kriging based spatiotemporal approach for traffic volume data imputation.

Along with the rapid development of Intelligent Transportation Systems, traffic data collection technologies have progressed fast. The emergence of innovative data collection technologies such as remote traffic microwave sensor, Bluetooth sensor, GPS-based floating car method, and automated license plate recognition, has significantly increased the variety and volume of traffic data. Despite the development of these technologies, the missing data issue is still a problem that poses great challenge for data based applications such as traffic forecasting, real-time incident detection, dynamic route guidance, and massive evacuation optimization. A thorough literature review suggests most current imputation models either focus on the temporal nature of the traffic data and fail to consider the spatial information of neighboring locations or assume the data follow a certain distribution. These two issues reduce the imputation accuracy and limit the use of the corresponding imputation methods respectively. As a result, this paper presents a Kriging based data imputation approach that is able to fully utilize the spatiotemporal correlation in the traffic data and that does not assume the data follow any distribution. A set of scenarios with different missing rates are used to evaluate the performance of the proposed method. The performance of the proposed method was compared with that of two other widely used methods, historical average and K-nearest neighborhood. Comparison results indicate that the proposed method has the highest imputation accuracy and is more flexible compared to other methods.

FROG: Inference from knowledge base for missing value imputation

Data imputation is a basic step for data cleaning. Traditional data imputation approaches are lack of accuracy in the absence of knowledge. Involving knowledge base in imputation could overcome this shortcoming. A challenge is that the missing value could be hardly found directly in the knowledge bases (KBs). To use knowledge base sufficiently for missing value imputation, we present FROG, an inference algorithm from knowledge bases. The inference not only makes full use of true facts in KBs, but also utilizes types to ensure the accuracy of captured missing values. Extensive experiments show that our proposed algorithm can capture missing values efficiently and effectively.

Evolutionary k-nearest neighbor imputation algorithm for gene expression data

Large data sets are produced by the gene expression process which is done by using the DNA microarray technology. These gene expression data are recognized as a common data source which contains missing expression values. In this paper, we present a genetic algorithm optimized k- Nearest neighbor algorithm (Evolutionary kNNImputation) for missing data imputation. Despite the common imputation methods this paper addresses the effectiveness of using supervised learning algorithms for missing data imputation. Missing data imputation approaches can be categorized into four main categories and among the four approaches, our focus is mainly on local approach where the proposed Evolutionary k- Nearest Neighbor Imputation Algorithm falls in. The Evolutionary k- Nearest Neighbor Imputation Algorithm is an extension of the common k- nearest Neighbor Imputation Algorithm which the genetic algorithm is used to optimize some parameters of k- Nearest Neighbor Algorithm. The selection of similarity matrix and the selection of the parameter value k can be identified as the optimization problem. We have compared the proposed Evolutionary k- Nearest Neighbor Imputation algorithm with k- Nearest Neighbor Imputation algorithm and mean imputation method. The three algorithms were tested using gene expression datasets. Certain percentages of values are randomly deleted in the datasets and recovered the missing values using the three algorithms. Results show that Evolutionary kNNImputation outperforms kNNImputation and mean imputation while showing the importance of using a supervised learning algorithm in missing data estimation. Even though mean imputation happened to show low mean error for a very few missing rates, supervised learning algorithms became effective when it comes to higher missing rates in datasets which is the most common situation among datasets.

Is structural stigma's effect on the mortality of sexual minorities robust? A failure to replicate the results of a published study

BackgroundThe study of stigma's influence on health has surged in recent years. Hatzenbuehler et al.’s (2014) study of structural stigma's effect on mortality revealed an average of 12 years' shorter life expectancy for sexual minorities who resided in communities thought to exhibit high levels of anti-gay prejudice, using data from the 1988–2002 administrations of the US General Social Survey linked to mortality outcome data in the 2008 National Death Index. MethodsIn the original study, the key predictor variable (structural stigma) led to results suggesting the profound negative influence of structural stigma on the mortality of sexual minorities. Attempts to replicate the study, in order to explore alternative hypotheses, repeatedly failed to generate the original study's key finding on structural stigma. Efforts to discern the source of the disparity in results revealed complications in the multiple imputation process for missing values of the components of structural stigma. This prompted efforts at replication using 10 different imputation approaches. ResultsEfforts to replicate Hatzenbuehler et al.’s (2014) key finding on structural stigma's notable influence on the premature mortality of sexual minorities, including a more refined imputation strategy than described in the original study, failed. No data imputation approach yielded parameters that supported the original study's conclusions. Alternative hypotheses, which originally motivated the present study, revealed little new information. ConclusionTen different approaches to multiple imputation of missing data yielded none in which the effect of structural stigma on the mortality of sexual minorities was statistically significant. Minimally, the original study's structural stigma variable (and hence its key result) is so sensitive to subjective measurement decisions as to be rendered unreliable.

Pre‐processing of incomplete spectrum sensing data in spectrum sensing data falsification attacks detection: a missing data imputation approach

Attacks detection is an important issue in collaborative spectrum sensing (CSS) system of cognitive radio networks. Many approaches are proposed to cope with the malicious behaviour of attackers. In existing works, spectrum sensing data (SSD) received by the fusion centre is generally assumed to be integrated. However, in practical scenarios, the received SSD may be incomplete due to the imperfect reporting channel or specific CSS schemes. The performance of existing attacks detection approaches may degrade especially when the probability of missing data is large. To alleviate this challenge, the authors focus on pre-processing of incomplete SSD and propose a practical imputation algorithm, which is derived from the maximum a posteriori probability criterion, to fill in the missing values. Simulation results indicate that the proposed algorithm restores the characteristics of the SSD, and mitigate the impacts of missing value on existing attacks detection algorithm effectively.

Dynamic Cost-Sensitive Extreme Learning Machine for Classification of Incomplete Data Based on the Deep Imputation Network

Due to its importance in many applications, the incomplete data mining has received increasing attention in recent years, but there has been little study of the cost-sensitive classification on incomplete data. Therefore this paper proposes the dynamic costsensitive extreme learning machine for classification of incomplete data based on the deep imputation network (DCELMIDC). Firstly, we propose an approach for incomplete data imputation based on the deep imputation network model, and offer the cost-sensitive extreme learning machine. Secondly, this paper introduces dynamic misclassification and test cost, and gives the chromosome coding and an evaluation method of the optimal cost. At last, on the basis of the genetic algorithm, the dynamic cost-sensitive extreme learning machine classification algorithm for mining incomplete data is given, which can search the optimal misclassification and test cost in cost spaces. The experiment results show that DCELMIDC is effective and feasible for classification of incomplete data, and can reduce the total cost.

Accuracy evaluation of different statistical and geostatistical censored data imputation approaches (Case study: Sari Gunay gold deposit)

Most of the geochemical datasets include missing data with different portions and this may cause a significant problem in geostatistical modeling or multivariate analysis of the data. Therefore, it is common to impute the missing data in most of geochemical studies. In this study, three approaches called half detection (HD), multiple imputation (MI), and the cosimulation based on Markov model 2 (MM2) are used to impute the censored data. According to the fact that the new datasets have to satisfy the original data underlying structure, the Multidimensional Scaling (MDS) approach has been used to explore the validity of different imputation methods. Log-ratio transformation (alr transformation) was performed to open the closed compositional data prior to applying the MDS method. Experiments showed that, based on the MDS approach, the MI and the MM2 could not satisfy the original underlying structure of the dataset as well as the HD approach. This is because these two mentioned approaches have produced values higher than the detection limit of the variables.

TRIP: An Interactive Retrieving-Inferring Data Imputation Approach

Data imputation aims at filling in missing attribute values in databases. Most existing imputation methods to string attribute values are inferring-based approaches, which usually fail to reach a high imputation recall by just inferring missing values from the complete part of the data set. Recently, some retrieving-based methods are proposed to retrieve missing values from external resources such as the World Wide Web, which tend to reach a much higher imputation recall, but inevitably bring a large overhead by issuing a large number of search queries. In this paper, we investigate the interaction between the inferring-based methods and the retrieving-based methods. We show that retrieving a small number of selected missing values can greatly improve the imputation recall of the inferring-based methods. With this intuition, we propose an inTeractive Retrieving-Inferring data imPutation approach (TRIP), which performs retrieving and inferring alternately in filling in missing attribute values in a data set. To ensure the high recall at the minimum cost, TRIP faces a challenge of selecting the least number of missing values for retrieving to maximize the number of inferable values. Our proposed solution is able to identify an optimal retrieving-inferring scheduling scheme in deterministic data imputation, and the optimality of the generated scheme is theoretically analyzed with proofs. We also analyze with an example that the optimal scheme is not feasible to be achieved in $\tau$ -constrained stochastic data imputation ( $\tau$ -SDI), but still, our proposed solution identifies an expected-optimal scheme in $\tau$ -SDI. Extensive experiments on four data collections show that TRIP retrieves on average 20 percent missing values and achieves the same high recall that was reached by the retrieving-based approach.

Overview of missing physical commodity trade data and its imputation using data augmentation

The physical aspects of commodity trade are becoming increasingly important on a global scale for transportation planning, demand management for transportation facilities and services, energy use, and environmental concerns. Such aspects (for example, weight and volume) of commodities are vital for logistics industry to allow for medium-to-long term planning at the strategic level and identify commodity flow trends. However, incomplete physical commodity trade databases impede proper analysis of trade flow between various countries. The missing physical values could be due to many reasons such as, (1) non-compliance of reporter countries with the prescribed regulations by World Customs Organization (WCO) (2) confidentiality issues, (3) delays in processing of data, or (4) erroneous reporting. The traditional missing data imputation methods, such as the substitution by mean, substitution by linear interpolation/extrapolation using adjacent points, the substitution by regression, and the substitution by stochastic regression, have been proposed in the context of estimating physical aspects of commodity trade data. However, a major demerit of these single imputation methods is their failure to incorporate uncertainty associated with missing data. The use of computationally complex stochastic methods to improve the accuracy of imputed data has recently become possible with the advancement of computer technology. Therefore, this study proposes a sophisticated data augmentation algorithm in order to impute missing physical commodity trade data. The key advantage of the proposed approach lies in the fact that instead of using a point estimate as the imputed value, it simulates a distribution of missing data through multiple imputations to reflect uncertainty and to maintain variability in the data. This approach also provides the flexibility to include fundamental distributional property of the variables, such as physical quantity, monetary value, price elasticity of demand, price variation, and product differentiation, and their correlations to generate reasonable average estimates of statistical inferences. An overview and limitations of most commonly used data imputation approaches is presented, followed by the theoretical basis and imputation procedure of the proposed approach. Lastly, a case study is presented to demonstrate the merits of the proposed approach in comparison to traditional imputation methods.

A wavelet-based thresholding approach to reconstructing unreliable spectrogram components

Data imputation approaches for robust automatic speech recognition reconstruct noise corrupted spectral information by exploiting prior knowledge of the relationship between target speech and background through the use of spectrographic masks. Most of these approaches are model-based techniques that can only provide accurate estimates of the underlying clean speech when the characteristics of the noise corrupted features do not deviate from those of the model. Discrete wavelet transform (DWT) based de-noising methods can also be used for re-estimating the underlying clean speech from a noise corrupted signal, but often require that the background noise is stationary and modeled by a Gaussian distribution. A novel approach is presented here for incorporating the information derived from spectrographic masks in a DWT-based de-noising method. The spectrographic masks are used for deriving thresholds for de-noising wavelet domain coefficients making DWT based de-noising more suitable for non-stationary noise conditions. The results of an experimental study are presented to demonstrate the performance of DWT based data imputation relative to other established techniques on the Aurora 2 noisy speech recognition task. It will be shown that the proposed approach reduces the impact of model mismatch associated with parametric approaches and exploits the robustness of non-parametric wavelet de-noising approach.

Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex‐ante and ex‐post peer evaluation data from the Austrian science fund

It is essential for research funding organizations to ensure both the validity and fairness of the grant approval procedure. The ex‐ante peer evaluation (EXANTE) of N = 8,496 grant applications submitted to the Austrian Science Fund from 1999 to 2009 was statistically analyzed. For 1,689 funded research projects an ex‐post peer evaluation (EXPOST) was also available; for the rest of the grant applications a multilevel missing data imputation approach was used to consider verification bias for the first time in peer‐review research. Without imputation, the predictive validity of EXANTE was low (r = .26) but underestimated due to verification bias, and with imputation it was r = .49. That is, the decision‐making procedure is capable of selecting the best research proposals for funding. In the EXANTE there were several potential biases (e.g., gender). With respect to the EXPOST there was only one real bias (discipline‐specific and year‐specific differential prediction). The novelty of this contribution is, first, the combining of theoretical concepts of validity and fairness with a missing data imputation approach to correct for verification bias and, second, multilevel modeling to test peer review‐based funding decisions for both validity and fairness in terms of potential and real biases.

A Nonparametric Multiple Imputation Approach for Data with Missing Covariate Values with Application to Colorectal Adenoma Data

A nearest neighbor-based multiple imputation approach is proposed to recover missing covariate information using the predictive covariates while estimating the association between the outcome and the covariates. To conduct the imputation, two working models are fitted to define an imputing set. This approach is expected to be robust to the underlying distribution of the data. We show in simulation and demonstrate on a colorectal data set that the proposed approach can improve efficiency and reduce bias in a situation with missing at random compared to the complete case analysis and the modified inverse probability weighted method.

Robust Missing Traffic Flow Imputation Considering Nonnegativity and Road Capacity

There are increasing concerns about missing traffic data in recent years. In this paper, a robust missing traffic flow data imputation approach based on matrix completion is proposed. In the proposed method, the similarity of traffic flow from day to day is exploited to impute missing data by the low-rank hypothesis of constructed traffic flow matrix. And the physical limitation of road capacity and nonnegativity is also considered through the optimization process, which avoids the possibility of producing negative and overcapacity values. Moreover, the proposed algorithm can impute missing data and recover outlier in a unify framework. The experiment results show that the proposed method is more accurate, stable, and reasonable.

Vehicle Reidentification with the Inductive Loop Signature Technology

This study proposed a real-time inductive loop signature based vehicle reidentification approach, RTREID-2M, which improved the previously developed RTREID-2 algorithm on two aspects: (1) Develop a cubic spline data imputation approach to replace the existing linear data imputation approach in order to improve the raw signature data quality; (2) Improve the time window setting for vehicles on High-Occupancy Vehicle (HOV) lanes so that vehicles on HOV lanes traveling with free flow speeds would be considered when generating candidate vehicle sets even during congestion time periods. In addition, a stratified-random sampling method was developed to effectively perform ground-truthing task for evaluating the performance of the proposed RTREID-2M. The evaluation results showed desired performance for vehicle reidentification and travel time estimation under both free-flow and congested flow traffic conditions. The future research will focus on the potential applications and arterial vehicle reidentification utilizing the inductive loop signature technologies.