Imputation Error Research Articles

NyuWa Genome resource: A deep whole-genome sequencing-based variation profile and reference panel for the Chinese population.

The lack of haplotype reference panels and whole-genome sequencing resources specific to the Chinese population has greatly hindered genetic studies in the world's largest population. Here, we present the NyuWa genome resource, based on deep (26.2×) sequencing of 2,999 Chinese individuals, and construct a NyuWa reference panel of 5,804 haplotypes and 19.3 million variants, which is a high-quality publicly available Chinese population-specific reference panel with thousands of samples. Compared with other panels, the NyuWa reference panel reduces the Han Chinese imputation error rate by a margin ranging from 30% to 51%. Population structure and imputation simulation tests support the applicability of one integrated reference panel for northern and southern Chinese. In addition, a total of 22,504 loss-of-function variants in coding and noncoding genes are identified, including 11,493 novel variants. These results highlight the value of the NyuWa genome resource in facilitating genetic research in Chinese and Asian populations.

Imputation of missing values for electronic health record laboratory data

Laboratory data from Electronic Health Records (EHR) are often used in prediction models where estimation bias and model performance from missingness can be mitigated using imputation methods. We demonstrate the utility of imputation in two real-world EHR-derived cohorts of ischemic stroke from Geisinger and of heart failure from Sutter Health to: (1) characterize the patterns of missingness in laboratory variables; (2) simulate two missing mechanisms, arbitrary and monotone; (3) compare cross-sectional and multi-level multivariate missing imputation algorithms applied to laboratory data; (4) assess whether incorporation of latent information, derived from comorbidity data, can improve the performance of the algorithms. The latter was based on a case study of hemoglobin A1c under a univariate missing imputation framework. Overall, the pattern of missingness in EHR laboratory variables was not at random and was highly associated with patients’ comorbidity data; and the multi-level imputation algorithm showed smaller imputation error than the cross-sectional method.

BiLSTM-I: A Deep Learning-Based Long Interval Gap-Filling Method for Meteorological Observation Data.

Complete and high-resolution temperature observation data are important input parameters for agrometeorological disaster monitoring and ecosystem modelling. Due to the limitation of field meteorological observation conditions, observation data are commonly missing, and an appropriate data imputation method is necessary in meteorological data applications. In this paper, we focus on filling long gaps in meteorological observation data at field sites. A deep learning-based model, BiLSTM-I, is proposed to impute missing half-hourly temperature observations with high accuracy by considering temperature observations obtained manually at a low frequency. An encoder-decoder structure is adopted by BiLSTM-I, which is conducive to fully learning the potential distribution pattern of data. In addition, the BiLSTM-I model error function incorporates the difference between the final estimates and true observations. Therefore, the error function evaluates the imputation results more directly, and the model convergence error and the imputation accuracy are directly related, thus ensuring that the imputation error can be minimized at the time the model converges. The experimental analysis results show that the BiLSTM-I model designed in this paper is superior to other methods. For a test set with a time interval gap of 30 days, or a time interval gap of 60 days, the root mean square errors (RMSEs) remain stable, indicating the model’s excellent generalization ability for different missing value gaps. Although the model is only applied to temperature data imputation in this study, it also has the potential to be applied to other meteorological dataset-filling scenarios.

Best practices for analyzing imputed genotypes from low-pass sequencing in dogs

Although DNA array-based approaches for genome-wide association studies (GWAS) permit the collection of thousands of low-cost genotypes, it is often at the expense of resolution and completeness, as SNP chip technologies are ultimately limited by SNPs chosen during array development. An alternative low-cost approach is low-pass whole genome sequencing (WGS) followed by imputation. Rather than relying on high levels of genotype confidence at a set of select loci, low-pass WGS and imputation rely on the combined information from millions of randomly sampled low-confidence genotypes. To investigate low-pass WGS and imputation in the dog, we assessed accuracy and performance by downsampling 97 high-coverage (> 15×) WGS datasets from 51 different breeds to approximately 1× coverage, simulating low-pass WGS. Using a reference panel of 676 dogs from 91 breeds, genotypes were imputed from the downsampled data and compared to a truth set of genotypes generated from high-coverage WGS. Using our truth set, we optimized a variant quality filtering strategy that retained approximately 80% of 14 M imputed sites and lowered the imputation error rate from 3.0% to 1.5%. Seven million sites remained with a MAF > 5% and an average imputation quality score of 0.95. Finally, we simulated the impact of imputation errors on outcomes for case–control GWAS, where small effect sizes were most impacted and medium-to-large effect sizes were minorly impacted. These analyses provide best practice guidelines for study design and data post-processing of low-pass WGS-imputed genotypes in dogs.

Consumption Imputation Errors in Administrative Data

Abstract Many research papers in household finance utilize annual snapshots of household wealth from administrative data, such as tax registries, to calculate “imputed consumption.” However, trading costs, unobserved intrayear trades, or unobserved security characteristics may cause measurement error. We document how such errors vary across groups of individuals by income, portfolio characteristics, and wealth and how they are correlated with individual income and balance sheets, asset prices, and the business cycle using transaction-level retail brokerage account data. We find that the economic significance of imputation error is small in many research settings, and we discuss robustness checks and econometric specifications to minimize the impact of imputation error in future research. Authors have furnished an Internet Appendix, which is available on the Oxford University Press Web site next to the link to the final published paper online.

Handling missing data in a rheumatoid arthritis registry using random forest approach

Missing data in clinical epidemiological research violate the intention-to-treat principle, reduce the power of statistical analysis, and can introduce bias if the cause of missing data is related to a patient's response to treatment. Multiple imputation provides a solution to predict the values of missing data. The main objective of this study is to estimate and impute missing values in patient records. The data from the Kuwait Registry for Rheumatic Diseases was used to deal with missing values among patient records. A number of methods were implemented to deal with missing data; however, choosing the best imputation method was judged by the lowest root mean square error (RMSE). Among 1735 rheumatoid arthritis patients, we found missing values vary from 5% to 65.5% of the total observations. The results show that sequential random forest method can estimate these missing values with a high level of accuracy. The RMSE varied between 2.5 and 5.0. missForest had the lowest imputation error for both continuous and categorical variables under each missing data rate (10%, 20%, and 30%) and had the smallest prediction error difference when the models used the imputed laboratory values.

Imputation methods for high-dimensional mixed-type datasets by nearest neighbors

In modern biomedical research, the data often contain a large number of variables of mixed data types (continuous, multi-categorical, or binary) but on some variables observations are missing. Imputation is a common solution when the downstream analyses require a complete data matrix. Several imputation methods are available that work under specific distributional assumptions. We propose an improvement over the popular non-parametric nearest neighbor imputation method which requires no particular assumptions. The proposed method makes practical and effective use of the information on the association among the variables. In particular, we propose a weighted version of the Lq distance for mixed-type data, which uses the information from a subset of important variables only. The performance of the proposed method is investigated using a variety of simulated and real data from different areas of application. The results show that the proposed methods yield smaller imputation error and better performance when compared to other approaches. It is also shown that the proposed imputation method works efficiently even when the number of samples is smaller than the number of variables.

PC-GAIN: Pseudo-label conditional generative adversarial imputation networks for incomplete data

Datasets with missing values are very common in real world applications. GAIN, a recently proposed deep generative model for missing data imputation, has been proved to outperform many state-of-the-art methods. But GAIN only uses a reconstruction loss in the generator to minimize the imputation error of the non-missing part, ignoring the potential category information which can reflect the relationship between samples. In this paper, we propose a novel unsupervised missing data imputation method named PC-GAIN, which utilizes potential category information to further enhance the imputation power. Specifically, we first propose a pre-training procedure to learn potential category information contained in a subset of low-missing-rate data. Then an auxiliary classifier is determined using the synthetic pseudo-labels. Further, this classifier is incorporated into the generative adversarial framework to help the generator to yield higher quality imputation results. The proposed method can improve the imputation quality of GAIN significantly. Experimental results on various benchmark datasets show that our method is also superior to other baseline approaches. Our code is available at https://github.com/WYu-Feng/pc-gain.

Using phylogenetic information to impute missing functional trait values in ecological databases

Trait-based approaches offer complementary views to the classic taxonomic approach, which is a crucial step forward to unveil mechanisms of community assembly, species interactions, ecosystem functioning, and tackling important conservation issues. These approaches require an enormous sampling effort to provide complete trait datasets, consequently, missing data are very common. We evaluated the performance of the missForest algorithm, that uses the Random Forest method to impute species traits values using phylogenetic information. We simulated datasets with different sizes and proportions of missing data, different levels of trait conservatism, and trait correlation. We tested trait imputation using the missForest algorithm without phylogenetic information and adding the phylogenetic relationship among species, using phylogenetic eigenvector. Our results show that the level of phylogenetic signal in traits and the correlation among them are the main parameters that influence the measures of imputation error. The measures of errors are smaller when traits have higher levels of correlation and when traits are conserved in the phylogenetic tree. In general, the inclusion of phylogenetic vectors as predictors in the missForest algorithm improves the estimation of missing values. However, the importance of phylogenetic information to the imputation process depends on the proportion of missing entries, phylogenetic conservatism of traits, and the correlation among traits. The missForest algorithm seems to be a robust method for trait imputation, and it can be used to estimate missing traits without the exclusion of species. Thus, we expect to have contributed with a new step to guide methodological choices to impute entire databases of traits with the goal of decreasing uncertainties and bias in the interpretation of ecological patterns and processes at different levels of ecological organization.

Assessing single-nucleotide polymorphism selection methods for the development of a low-density panel optimized for imputation in South African Drakensberger beef cattle.

A major obstacle in applying genomic selection (GS) to uniquely adapted local breeds in less-developed countries has been the cost of genotyping at high densities of single-nucleotide polymorphisms (SNP). Cost reduction can be achieved by imputing genotypes from lower to higher densities. Locally adapted breeds tend to be admixed and exhibit a high degree of genomic heterogeneity thus necessitating the optimization of SNP selection for downstream imputation. The aim of this study was to quantify the achievable imputation accuracy for a sample of 1,135 South African (SA) Drakensberger cattle using several custom-derived lower-density panels varying in both SNP density and how the SNP were selected. From a pool of 120,608 genotyped SNP, subsets of SNP were chosen (1) at random, (2) with even genomic dispersion, (3) by maximizing the mean minor allele frequency (MAF), (4) using a combined score of MAF and linkage disequilibrium (LD), (5) using a partitioning-around-medoids (PAM) algorithm, and finally (6) using a hierarchical LD-based clustering algorithm. Imputation accuracy to higher density improved as SNP density increased; animal-wise imputation accuracy defined as the within-animal correlation between the imputed and actual alleles ranged from 0.625 to 0.990 when 2,500 randomly selected SNP were chosen vs. a range of 0.918 to 0.999 when 50,000 randomly selected SNP were used. At a panel density of 10,000 SNP, the mean (standard deviation) animal-wise allele concordance rate was 0.976 (0.018) vs. 0.982 (0.014) when the worst (i.e., random) as opposed to the best (i.e., combination of MAF and LD) SNP selection strategy was employed. A difference of 0.071 units was observed between the mean correlation-based accuracy of imputed SNP categorized as low (0.01 < MAF ≤ 0.1) vs. high MAF (0.4 < MAF ≤ 0.5). Greater mean imputation accuracy was achieved for SNP located on autosomal extremes when these regions were populated with more SNP. The presented results suggested that genotype imputation can be a practical cost-saving strategy for indigenous breeds such as the SA Drakensberger. Based on the results, a genotyping panel consisting of ~10,000 SNP selected based on a combination of MAF and LD would suffice in achieving a <3% imputation error rate for a breed characterized by genomic admixture on the condition that these SNP are selected based on breed-specific selection criteria.

An efficient ensemble method for missing value imputation in microarray gene expression data

BackgroundThe genomics data analysis has been widely used to study disease genes and drug targets. However, the existence of missing values in genomics datasets poses a significant problem, which severely hinders the use of genomics data. Current imputation methods based on a single learner often explores less known genomic data information for imputation and thus causes the imputation performance loss.ResultsIn this study, multiple single imputation methods are combined into an imputation method by ensemble learning. In the ensemble method, the bootstrap sampling is applied for predictions of missing values by each component method, and these predictions are weighted and summed to produce the final prediction. The optimal weights are learned from known gene data in the sense of minimizing a cost function about the imputation error. And the expression of the optimal weights is derived in closed form. Additionally, the performance of the ensemble method is analytically investigated, in terms of the sum of squared regression errors. The proposed method is simulated on several typical genomic datasets and compared with the state-of-the-art imputation methods at different noise levels, sample sizes and data missing rates. Experimental results show that the proposed method achieves the improved imputation performance in terms of the imputation accuracy, robustness and generalization.ConclusionThe ensemble method possesses the superior imputation performance since it can make use of known data information more efficiently for missing data imputation by integrating diverse imputation methods and learning the integration weights in a data-driven way.

Tensor Data Imputation by PARAFAC with Updated Chaotic Biases by Adam Optimizer

The big data pattern analysis suffers from incorrect responses due to missing data entries in the real world. Data collected for digital movie platforms like Netflix and intelligent transportation systems is Spatio-temporal data. Extracting the latent and explicit features from this data is a challenge. We present the high dimensional data imputation problem as a higher-order tensor decomposition. The regularized and biased PARAFAC decomposition is proposed to generate the missing data entries. The biases are created and updated by a chaotic exponential factor in Adam’s optimization, which reduces the imputation error. This chaotic perturbed exponentially update in the learning rate replaces the fixed learning rate in the bias update by Adam optimization. The idea has experimented with Netflix and traffic datasets from Guangzhou, China.

Handling Complex Missing Data Using Random Forest Approach for an Air Quality Monitoring Dataset: A Case Study of Kuwait Environmental Data (2012 to 2018)

In environmental research, missing data are often a challenge for statistical modeling. This paper addressed some advanced techniques to deal with missing values in a data set measuring air quality using a multiple imputation (MI) approach. MCAR, MAR, and NMAR missing data techniques are applied to the data set. Five missing data levels are considered: 5%, 10%, 20%, 30%, and 40%. The imputation method used in this paper is an iterative imputation method, missForest, which is related to the random forest approach. Air quality data sets were gathered from five monitoring stations in Kuwait, aggregated to a daily basis. Logarithm transformation was carried out for all pollutant data, in order to normalize their distributions and to minimize skewness. We found high levels of missing values for (18.4%), (18.5%), (57.4%), (19.0%), and (18.2%) data. Climatological data (i.e., air temperature, relative humidity, wind direction, and wind speed) were used as control variables for better estimation. The results show that the MAR technique had the lowest RMSE and MAE. We conclude that MI using the missForest approach has a high level of accuracy in estimating missing values. MissForest had the lowest imputation error (RMSE and MAE) among the other imputation methods and, thus, can be considered to be appropriate for analyzing air quality data.

Nonlinear compensation algorithm for multidimensional temporal data: A missing value imputation for the power grid applications

In smart grid, the missing values do influence the real-time grid monitoring and bring biases of conclusions from the grid data mining. From the analysis on the data from smart grid, every variable shows global variation and local variation. Based on these characters, a novel statistical and machine learning-based imputation method is proposed, taking advantage of the global trend capturing by one-dimension interpolation of the variable of interest and the local variation capturing by linear compensation of multidimensional variables. By using KCPA, the multidimensional nonlinear variables are mapped into a feature space, and obtained new variables linearly couple with the variable of interest. Then these new variables together with the multidimensional linear variables are used for that linear compensation. The comparative experiment indicates that the proposed method outperforms the commonly used methods by reducing the RMSE by 29.19% and MAE by 44.73% on average, and having the best R2 closest to 1. A test on public dataset shows that the proposed method still has a good performance. At last, the sensitivity analysis on missing rate shows that the imputation error of the proposed methods remains steady for all the variables with the increase of missing rates from 5% to 10%.

Active matrix factorization for surveys

Amid historically low response rates, survey researchers seek ways to reduce respondent burden while measuring desired concepts with precision. We propose to ask fewer questions of respondents and impute missing responses via probabilistic matrix factorization. A variance-minimizing active learning criterion chooses the most informative questions per respondent. In simulations of our matrix sampling procedure on real-world surveys as well as a Facebook survey experiment, we find active question selection achieves efficiency gains over baselines. The reduction in imputation error is heterogeneous across questions and depends on the latent concepts they capture. Modeling responses with the ordered logit likelihood improves imputations and yields an adaptive question order. We find for the Facebook survey that potential biases from order effects are likely to be small. With our method, survey researchers obtain principled suggestions of questions to retain and, if desired, can automate the design of shorter instruments.

Increasing accuracy of HLA imputation by a population-specific reference panel in a FinnGen biobank cohort.

The HLA genes, the most polymorphic genes in the human genome, constitute the strongest single genetic susceptibility factor for autoimmune diseases, transplantation alloimmunity and infections. HLA imputation via statistical inference of alleles based on single-nucleotide polymorphisms (SNPs) in linkage disequilibrium (LD) with alleles is a powerful first-step screening tool. Due to different LD structures between populations, the accuracy of HLA imputation may benefit from matching the imputation reference with the study population. To evaluate the potential advantage of using population-specific reference in HLA imputation, we constructed an HLA reference panel consisting of 1150 Finns with 5365 major histocompatibility complex region SNPs consistent between genome builds. We evaluated the accuracy of the panel against a European panel in an independent test set of 213 Finnish subjects. We show that the Finnish panel yields a lower imputation error rate (1.24% versus 1.79%). More than 30% of imputation errors occurred in haplotypes enriched in Finland. The frequencies of imputed HLA alleles were highly correlated with clinical-grade HLA allele frequencies and allowed accurate replication of established HLA–disease associations in ∼102 000 biobank participants. The results show that a population-specific reference increases imputation accuracy in a relatively isolated population within Europe and can be successfully applied to biobank-scale genome data collections.

Fuzzy case‐based‐reasoning‐based imputation for incomplete data in software engineering repositories

AbstractMissing data is a serious issue in software engineering because it can lead to information loss and bias in data analysis. Several imputation techniques have been proposed to deal with both numerical and categorical missing data. However, most of those techniques used is simple reuse techniques originally designed for numerical data, which is a problem when the missing data are related to categorical attributes. This paper aims (a) to propose a new fuzzy case‐based reasoning (CBR) imputation technique designed for both numerical and categorical data and (b) to evaluate and compare the performance of the proposed technique with the k‐nearest neighbor (KNN) imputation technique in terms of error and accuracy under different missing data percentages and missingness mechanisms in four software engineering data sets. The results suggest that the proposed fuzzy CBR technique outperformed KNN in terms of imputation error and accuracy regardless of the missing data percentage, missingness mechanism, and data set used. Moreover, we found that the missingness mechanism has an important impact on the performance of both techniques. The results are encouraging in the sense that using an imputation technique designed for both categorical and numerical data is better than reusing methods originally designed for numerical data.

Incomplete Road Information Imputation Using Parallel Interpolation to Enhance the Safety of Autonomous Driving

Autonomous driving is the main application of Internet of Things (IoT) technology in the field of intelligent transportation. In autonomous driving, self-driving cars will avoid changing lanes in a short distance. When the self-driving car executes the follow-up instruction, the road smoothness in front of the car will affect the driving safety and comfort of the car. The real-time acquisition of road information in front of driving will help the self-driving car adjust driving behavior. However, other vehicles on the road will lead to the failure of Light Detection And Ranging (LiDAR) detectors to obtain complete road point cloud data. The incomplete road point cloud data need to be imputated to avoid potential misjudgements of the road conditions. Currently, little research work specifically focuses on imputating the incomplete road point cloud data that are caused by obstacle vehicles. In this paper, we propose a fast method to imputate the incomplete road point cloud data using a Graphics Processing Unit (GPU)-based parallel Inverse Distance Weighted (IDW) interpolation algorithm to enhance the safety of autonomous driving. To evaluate the performance of the proposed method, two groups of experiments are conducted. The experimental results indicate the following: (1) the known point cloud data within 5 meters around the obstacle vehicle are sufficient to guarantee the imputation accuracy; (2) when the weight parameter of the IDW interpolation is 4, the efficiency and accuracy of the imputation can be optimally balanced; and (3) it takes approximately 0.6 seconds to imputate the incomplete dataset consisting of 15 million points, while the imputation error is approximately 5 millimeters. The proposed method is capable of efficiently and effectively imputating the incomplete road point cloud data that are induced by obstacle vehicles and outperforms other interpolation algorithms and machine learning algorithms.

Refinements of LC-MS/MS Spectral Counting Statistics Improve Quantification of Low Abundance Proteins

Mass spectrometry-based spectral count has been a common choice of label-free proteome quantification due to the simplicity for the sample preparation and data generation. The discriminatory nature of spectral count in the MS data-dependent acquisition, however, inherently introduces the spectral count variation for low-abundance proteins in multiplicative LC-MS/MS analysis, which hampers sensitive proteome quantification. As many low-abundance proteins play important roles in cellular processes, deducing low-abundance proteins in a quantitatively reliable manner greatly expands the depth of biological insights. Here, we implemented the Moment Adjusted Imputation error model in the spectral count refinement as a post PLGEM-STN for improving sensitivity for quantitation of low-abundance proteins by reducing spectral count variability. The statistical framework, automated spectral count refinement by integrating the two statistical tools, was tested with LC-MS/MS datasets of MDA-MB468 breast cancer cells grown under normal and glucose deprivation conditions. We identified about 30% more quantifiable proteins that were found to be low-abundance proteins, which were initially filtered out by the PLGEM-STN analysis. This newly developed statistical framework provides a reliable abundance measurement of low-abundance proteins in the spectral count-based label-free proteome quantification and enabled us to detect low-abundance proteins that could be functionally important in cellular processes.

Impacts of SNP genotyping call rate and SNP genotyping error rate on imputation accuracy inHolsteincattle

Single nucleotide polymorphism (SNP) chips have been widely used in genetic studies and breeding applications in animal and plant species. The quality of SNP genotypes is of paramount importance. More often than not, there are situations in which a number of genotypes may fail, requiring them to be imputed. There are also situations in which ungenotyped loci need to be imputed between different chips, or high-density genotypes need to be imputed based on low-density genotypes. Under these circumstances, the validity and reliability of subsequent data analyses is subject to the accuracy of these imputed genotypes. For justifying a better understanding of factors affecting imputation accuracy, in the present study, the impacts of SNP genotyping call rate and SNP genotyping error rate on the accuracy of genotype imputation were investigated under two scenarios in 20 116 U.S. Holstein cattle, each genotyped with a GGP 50K SNP chip. When the two factors were not correlated in scenario 1, simulated genotyping call rate varied from 50% to 100% and simulated genotyping error rate changed from 0% to 50%, with both factors being independent of each other. In scenario 2, genotyping error rates were correlated with genotyping call rate, and the relationship was set up by fitting a linear regression model between the two variables on a real dataset. That is, the simulated SNP call rate varied from 100% to 50% whereas the SNP genotyping rate changed from 0% to 13.55%. Finally, a 5-fold cross-validation was used to assess the subsequent imputation accuracy. The results showed that when original SNP genotyping call rate were independent of SNP genotyping error rate, the imputation accuracy did not change significantly with the original genotyping call rate (P>0.05), but it decreased significantly as the genotyping error rate increased (P<0.01). However, when original genotyping call rate was negatively correlated with genotyping error rate, the imputation error increased with elevated original genotyping error rate. In both scenarios, genotyping call rate needs to be no less than 0.90 in order to obtain 98% or higher genotype imputation accuracy. The present results can provide guidance for establishing quality assurance criteria for SNP genotyping in practice.