Value Imputation Research Articles

Selective Imputation of Covariates in High Dimensional Censored Data

Efficient modeling of censored data, that is, data which are restricted by some detection limit or truncation, is important for many applications. Ignoring the censoring can be problematic as valuable information may be missing and restoration of these censored values may significantly improve the quality of models. There are many scenarios where one may encounter censored data: survival data, interval-censored data or data with a lower limit of detection. Strategies to handle censored data are plenty, however, little effort has been made to handle censored data of high dimension. In this article, we present a selective multiple imputation approach for predictive modeling when a larger number of covariates are subject to censoring. Our method allows for iterative, subject-wise selection of covariates to impute in order to achieve a fast and accurate predictive model. The algorithm furthermore selects values for imputation which are likely to provide important information if imputed. In contrast to previously proposed methods, our approach is fully nonparametric and therefore, very flexible. We demonstrate that, in comparison to previous work, our model achieves faster execution and often comparable accuracy in a simulated example as well as predicting signal strength in radio network data. Supplementary materials for this article are available online.

Cancer Patients Missing Pain Score Information:- Application with Imputation Techniques

Background: Methods for handling missing data in clinical research have been getting more attentions over last few years. Contemplation of missing data in any study is vital as they may lead to considerable biases and can have an impact on the power of the study.
 Objective: This manuscript is dedicated to present different techniques to handle missing observations obtained from repeatedly measured pain score data on palliative cancer.
 Methods: This problem caused by subjects drop out before completion of the study. The reason for dropout or withdrawal may be related to study (e.g., adverse event, death, unpleasant study procedures, lack of improvement) or unrelated to the study (e.g., moving away, unrelated disease). The dropout might be very common on studies on palliative cancer patients. The Palliative treatment is designed to relieve symptoms, and improve the quality of life and can be used at any stage of an illness if there are troubling symptoms, such as pain or sickness.
 Results: The mean(SD) of observed pain score was 3.638(3.156) whereas the imputed mean values were 3.615(2.980), 3.618(2.954), 3.577(2.892), 3.560(2.999) and 3.627(2.949) respectively for the imputation methods regression, predictive mean matching, propensity score, EM algorithm and MCMC methods for pain score values at visit3. 
 Interpretation and Conclusion: The EM algorithm shows the least percentage change from observed values in both visits followed by predictive mean matching method and MCMC methods. The multiple imputation techniques have few advantages; the imputed values are drawsfrom a distribution, so they inherently contain some variation by introducing an additional form of error in the parameter estimates across the imputation

Factor-based imputation of missing values and covariances in panel data of large dimensions

Economists are blessed with a wealth of data for analysis, but more often than not, values in some entries of the data matrix are missing. Various methods have been proposed to handle missing observations in a few variables. We exploit the factor structure in panel data of large dimensions. Our tall-project algorithm first estimates the factors from a tall block in which data for all rows are observed, and projections of unit specific sample size are then used to estimate the factor loadings. A missing value is imputed by its estimated common component which we show is consistent and asymptotically normal without further iteration. Implications for using imputed data in factor augmented regressions are then discussed. To compensate for the downward bias in sample covariance matrices created by an omitted noise in each imputed value, we overlay the imputed data with re-sampled idiosyncratic residuals many times and use the average of the covariances to estimate the parameters of interest. Simulations show that the procedures have desirable finite sample properties.

PARTITIONING AROUND MEDOIDS CLUSTERING AND RANDOM FOREST CLASSIFICATION FOR GIS-INFORMED IMPUTATION OF FLUORIDE CONCENTRATION DATA.

Community water fluoridation is an important component of oral health promotion, as fluoride exposure is a well-documented dental caries-preventive agent. Direct measurements of domestic water fluoride content provide valuable information regarding individuals' fluoride exposure and thus caries risk; however, they are logistically challenging to carry out at a large scale in oral health research. This article describes the development and evaluation of a novel method for the imputation of missing domestic water fluoride concentration data informed by spatial autocorrelation. The context is a state-wide epidemiologic study of pediatric oral health in North Carolina, where domestic water fluoride concentration information was missing for approximately 75% of study participants with clinical data on dental caries. A new machine-learning-based imputation method that combines partitioning around medoids clustering and random forest classification (PAMRF) is developed and implemented. Imputed values are filtered according to allowable error rates or target sample size, depending on the requirements of each application. In leave-one-out cross-validation and simulation studies, PAMRF outperforms four existing imputation approaches-two conventional spatial interpolation methods (i.e., inverse-distance weighting, IDW and universal kriging, UK) and two supervised learning methods (k-nearest neighbors, KNN and classification and regression trees, CART). The inclusion of multiply imputed values in the estimation of the association between fluoride concentration and dental caries prevalence resulted in essentially no change in PAMRF estimates but substantial gains in precision due to larger effective sample size. PAMRF is a powerful new method for the imputation of missing fluoride values where geographical information exists.

Increased transparency was required when reporting imputation of primary outcome data in clinical trials

ObjectivesThe aim of this study is to explore the transparency of reporting primary outcome data within randomized controlled trials in the presence of missing data. Study Design and SettingA cohort examination of randomized controlled trials published in the four major medical journals (NEJM, JAMA, BMJ, Lancet) in 2013 and the first quarter of 2018. Data were extracted on reporting quality, the number of randomized participants, and the number of participants included within the primary outcome analysis with observed or imputed data. ResultsNinety-one of 159 (57%) studies analyzed from 2013 and 19 of 46 (41%) from 2018 included imputed data within the primary outcome analysis. Of these, only 13 of 91 (14%) studies from 2013 and 1 of 19 (5%) studies from 2018 explicitly reported the number of imputed values in the CONSORT diagram. Results’ tables included levels of imputed data in 12 of 91 (13%) studies in 2013 and 4 of 19 (21%) in 2018. Consequently, identification of imputed data was a time-consuming task requiring extensive cross-referencing of all manuscript elements. ConclusionImputed primary outcome data are poorly reported. Participant flow diagrams frequently reported participant status which does not necessarily correlate to availability of data. We recommended that the number of imputed values are explicitly reported within CONSORT flow diagrams to increase transparency.

A Rescaling Bootstrap Approach For Imputed Survey Data

Abstract Imputation is usually used to deal with item nonresponse in surveys. Treating the imputed values as true observations may obviously lead to serious underestimation of the variance of point estimators. In this article, we propose a new bootstrap method under the rescaling bootstrap approach for estimating the variance of an imputed estimator obtained after applying deterministic regression or random hot-deck imputation. A novel technique is used to rescale the original data set through solving certain systems of linear equations. The proposed procedure can handle unequal response probabilities and large sampling fractions. Some simulation studies are conducted to show the great performance of the proposed method in terms of relative bias, relative efficiency, and coverage probability, for both population mean and median.

Improving Penalized Logistic Regression Model with Missing Values in High-Dimensional Data

Analysis without adequate handling of missing values may lead to inconsistent and biased estimates. Despite multiple imputations becoming a widely used approach in handling missing data, manuscript researchers generally encounter missing data in their respective studies. In high-dimensional data, penalized regression is a popular technique for performing feature selection and coefficient estimation simultaneously. However, one of the most vital issues with high-dimensional data is that it often contains large quantities of missing data that common multiple imputation approaches may not work correctly. Therefore, this study uses imputations penalized regression models as an extension of the penalized methods to improve the performance and impute missing values in high-dimensional data. The method was applied to real-life high-dimensional datasets for the different number of features, sample sizes, and missing dataset rates to evaluate its efficiency. The method was also compared with other existing imputation penalized methods for high-dimensional data. The comparative experimental results indicate that the proposed method outperforms its competitors by achieving higher sensitivity, specificity, and classification accuracy values.

A reinforcement learning-based approach for imputing missing data

Missing data is a major problem in real-world datasets, which hinders the performance of data analytics. Conventional data imputation schemes such as univariate single imputation replace missing values in each column with the same approximated value. These univariate single imputation techniques underestimate the variance of the imputed values. On the other hand, multivariate imputation explores the relationships between different columns of data, to impute the missing values. Reinforcement Learning (RL) is a machine learning paradigm where the agent learns by taking actions and receiving rewards in response, to achieve its goal. In this work, we propose an RL-based approach to impute missing data by learning a policy to impute data through an action-reward-based experience. Our approach imputes missing values in a column by working only on the same column (similar to univariate single imputation) but imputes the missing values in the column with different values thus keeping the variance in the imputed values. We report superior performance of our approach, compared with other imputation techniques, on a number of datasets.

Nonresponse bias in change estimation: a national forest inventory example

Abstract Nonresponse in national forest inventories primarily occurs in forested areas due to accessibility issues or where hazardous conditions exist. As with all surveys, nonresponse has the potential to impart empirical bias into sample-based estimates and care should be taken to minimize any effects. A less-studied aspect is the effects of nonresponse when estimating change between two points in time. In this study, potential bias in change estimates was evaluated using imputed values for nonresponse inventory plots to compare differences between response and nonresponse means. Analysis of forest area and tree biomass density attributes revealed that systematic differences in probabilities of nonresponse that occur due to ownership type and forest/nonforest status produce overall estimates of change that are too small. The empirical bias appears to worsen as nonresponse rates increase. Underestimation of change inhibits detection of statistically significant shifts in forest resource attributes and concurrently thwarts effective management and policy responses. Thus, further study to ameliorate this issue is warranted, including improved strategies for defining populations and strata to better conform to nonresponse assumptions and/or alternative estimation methods that account for differential nonresponse probabilities due to ownership or forest/nonforest status.

Tackling UCR's missing data problem: A multiple imputation approach

Purpose: This paper addresses concerns related to missing data in the UCR. We compare several methods using multiple imputation in an effort to improve the data. Methods: We develop a two-stage procedure for imputing missing values in the UCR. First, we use agencies' past reporting to impute missing values. Then, after merging in demographic data from the census, we use multiple imputation to fill in the remaining missing values. Several methods are considered, including multivariate normal imputation (MVNI) and multiple imputation by chained equations (MICE). Results: Methods are compared using a complete subset of the UCR with values systematically deleted. We find that a model using MICE with random forest prediction produces results that are comparable to the methods NACJD uses when preparing the county level UCR data. Conclusions: We recommend applying multiple imputation to the UCR rather than relying on the county level data published by NACJD. Not only are our results similar to NACJD's, but additionally, an approach using multiple imputation accounts for the uncertainty in imputed values, so that correct standard errors can be obtained in subsequent analyses.

Multiple imputation for longitudinal data using Bayesian lasso imputation model.

Multiple imputation is a promising approach to handle missing data and is widely used in analysis of longitudinal clinical studies. A key consideration in the implementation of multiple imputation is to obtain accurate imputed values by specifying an imputation model that incorporates auxiliary variables potentially associated with missing variables. The use of informative auxiliary variables is known to be beneficial to make the missing at random assumption more plausible and help to reduce uncertainty of the imputations; however, it is not straightforward to pre-specify them in many cases. We propose a data-driven specification of the imputation model using Bayesian lasso in the context of longitudinal clinical study, and develop a built-in function of the Bayesian lasso imputation model which is performed within the framework of multiple imputation using chained equations. A simulation study suggested that the Bayesian lasso imputation model worked well in a variety of longitudinal study settings, providing unbiased treatment effect estimates with well-controlled type I error rates and coverage probabilities of the confidence interval; in contrast, ignorance of the informative auxiliary variables led to serious bias and inflation of type I error rate. Moreover, the Bayesian lasso imputation model offered higher statistical powers compared with conventional imputation methods. In our simulation study, the gains in statistical power were remarkable when the sample size was small relative to the number of auxiliary variables. An illustration through a real example also suggested that the Bayesian lasso imputation model could give smaller standard errors of the treatment effect estimate.

Imputation methods for missing response values in the three parts of a central composite design with two factors

This study compares four methods (Mean, Part-mean, Regression, and K-nearest neighbor (KNN)) for imputing the missing response values in a CCD. Four test functions are used to perform all possible cases of a single missing response in a CCD with two factors. The performance was measured by using the mean-squared error and correlation coefficient of the three parts of the CCD (factorial, center, or axial) by comparing the imputed and actual values. The study results show that the missing response value's influence of the affected part of the CCD (factorial, center, axial) could not be neglected, and the Regression imputation method was superior to the other three for an imputed value in the factorial or axial parts of the CCD. Furthermore, for missing values in the center part, the results show that the Part-mean imputation method was equally as good as the complex imputation methods of Regression and KNN.

Application of machine learning missing data imputation techniques in clinical decision making: taking the discharge assessment of patients with spontaneous supratentorial intracerebral hemorrhage as an example

BackgroundThere are often many missing values in medical data, which directly affect the accuracy of clinical decision making. Discharge assessment is an important part of clinical decision making. Taking the discharge assessment of patients with spontaneous supratentorial intracerebral hemorrhage as an example, this study adopted the missing data processing evaluation criteria more suitable for clinical decision making, aiming at systematically exploring the performance and applicability of single machine learning algorithms and ensemble learning (EL) under different data missing scenarios, as well as whether they had more advantages than traditional methods, so as to provide basis and reference for the selection of suitable missing data processing method in practical clinical decision making.MethodsThe whole process consisted of four main steps: (1) Based on the original complete data set, missing data was generated by simulation under different missing scenarios (missing mechanisms, missing proportions and ratios of missing proportions of each group). (2) Machine learning and traditional methods (eight methods in total) were applied to impute missing values. (3) The performances of imputation techniques were evaluated and compared by estimating the sensitivity, AUC and Kappa values of prediction models. (4) Statistical tests were used to evaluate whether the observed performance differences were statistically significant.ResultsThe performances of missing data processing methods were different to a certain extent in different missing scenarios. On the whole, machine learning had better imputation performance than traditional methods, especially in scenarios with high missing proportions. Compared with single machine learning algorithms, the performance of EL was more prominent, followed by neural networks. Meanwhile, EL was most suitable for missing imputation under MAR (the ratio of missing proportion 2:1) mechanism, and its average sensitivity, AUC and Kappa values reached 0.908, 0.924 and 0.596 respectively.ConclusionsIn clinical decision making, the characteristics of missing data should be actively explored before formulating missing data processing strategies. The outstanding imputation performance of machine learning methods, especially EL, shed light on the development of missing data processing technology, and provided methodological support for clinical decision making in presence of incomplete data.

A data quality assessment of the first four years of malaria reporting in the Senegal DHIS2, 2014\u20132017

BackgroundAs the global burden of malaria decreases, routine health information systems (RHIS) have become invaluable for monitoring progress towards elimination. The District Health Information System, version 2 (DHIS2) has been widely adopted across countries and is expected to increase the quality of reporting of RHIS. In this study, we evaluated the quality of reporting of key indicators of childhood malaria from January 2014 through December 2017, the first 4 years of DHIS2 implementation in Senegal.MethodsMonthly data on the number of confirmed and suspected malaria cases as well as tests done were extracted from the Senegal DHIS2. Reporting completeness was measured as the number of monthly reports received divided by the expected number of reports in a given year. Completeness of indicator data was measured as the percentage of non-missing indicator values. We used a quasi-Poisson model with natural cubic spline terms of month of reporting to impute values missing at the facility level. We used the imputed values to take into account the percentage of malaria cases that were missed due to lack of reporting. Consistency was measured as the absence of moderate and extreme outliers, internal consistency between related indicators, and consistency of indicators over time.ResultsIn contrast to public facilities of which 92.7% reported data in the DHIS2 system during the study period, only 15.3% of the private facilities used the reporting system. At the national level, completeness of facility reporting increased from 84.5% in 2014 to 97.5% in 2017. The percentage of expected malaria cases reported increased from 76.5% in 2014 to 94.7% in 2017. Over the study period, the percentage of malaria cases reported across all districts was on average 7.5% higher (P < 0.01) during the rainy season relative to the dry season. Reporting completeness rates were lower among hospitals compared to health centers and health posts. The incidence of moderate and extreme outlier values was 5.2 and 2.3%, respectively. The number of confirmed malaria cases increased by 15% whereas the numbers of suspected cases and tests conducted more than doubled from 2014 to 2017 likely due to a policy shift towards universal testing of pediatric febrile cases.ConclusionsThe quality of reporting for malaria indicators in the Senegal DHIS2 has improved over time and the data are suitable for use to monitor progress in malaria programs, with an understanding of their limitations. Senegalese health authorities should maintain the focus on broader adoption of DHIS2 reporting by private facilities, the sustainability of district-level data quality reviews, facility-level supervision and feedback mechanisms at all levels of the health system.

Effects of hybrid non-linear feature extraction method on different data sampling techniques for liver disease prediction

Liver disease indicates inflammatory condition of the liver, liver cirrhosis, cancer, or an overload of toxic substances. A liver transplant may reinstate and extend life if a patient has severe liver disease. In the last few years, machine learning (ML) based diagnosis systems have played a vital role in assessing liver patients which eventually leads to proper treatment and saves human life. In this study, we try to predict liver patients by adopting a hybrid feature extraction method to enhance the performance of the ML algorithm. Medical data frequently exhibits non-linear patterns and class imbalances. This is undesirable for the majority of ML algorithms and degrades performance. Here, we present a hybrid feature space that combines t-SNE, Isomap nonlinear features, and kernel principal components that can explain 90% of the variation in the data as a solution to this issue. Before feeding the ML model, data preprocessing techniques including class balancing, identifying outliers, and impute missing values are used. A simulation study and ensemble learning also conducted to justify the proposed prediction performances. Our suggested hybrid non-linear feature exhibits a 2-20 % improvement over existing studies and the ensemble classifier achieved an ideal and outstanding accuracy of 91.33 %.

Evaluating the Use of Uncertainty Visualisations for Imputations of Data Missing At Random in Scatterplots.

Most real-world datasets contain missing values yet most exploratory data analysis (EDA) systems only support visualising data points with complete cases. This omission may potentially lead the user to biased analyses and insights. Imputation techniques can help estimate the value of a missing data point, but introduces additional uncertainty. In this work, we investigate the effects of visualising imputed values in charts using different ways of representing data imputations and imputation uncertainty-no imputation, mean, 95% confidence intervals, probability density plots, gradient intervals, and hypothetical outcome plots. We focus on scatterplots, which is a commonly used chart type, and conduct a crowdsourced study with 202 participants. We measure users' bias and precision in performing two tasks-estimating average and detecting trend-and their self-reported confidence in performing these tasks. Our results suggest that, when estimating averages, uncertainty representations may reduce bias but at the cost of decreasing precision. When estimating trend, only hypothetical outcome plots may lead to a small probability of reducing bias while increasing precision. Participants in every uncertainty representation were less certain about their response when compared to the baseline. The findings point towards potential trade-offs in using uncertainty encodings for datasets with a large number of missing values. This paper and the associated analysis materials are available at: https://osf.io/q4y5r/.

Crop loss identification at field parcel scale using satellite remote sensing and machine learning

Identifying crop loss at field parcel scale using satellite images is challenging: first, crop loss is caused by many factors during the growing season; second, reliable reference data about crop loss are lacking; third, there are many ways to define crop loss. This study investigates the feasibility of using satellite images to train machine learning (ML) models to classify agricultural field parcels into those with and without crop loss. The reference data for this study was provided by Finnish Food Authority (FFA) containing crop loss information of approximately 1.4 million field parcels in Finland covering about 3.5 million ha from 2000 to 2015. This reference data was combined with Normalised Difference Vegetation Index (NDVI) derived from Landsat 7 images, in which more than 80% of the possible data are missing. Despite the hard problem with extremely noisy data, among the four ML models we tested, random forest (with mean imputation and missing value indicators) achieved the average AUC (area under the ROC curve) of 0.688±0.059 over all 16 years with the range [0.602, 0.795] in identifying new crop-loss fields based on reference fields of the same year. To our knowledge, this is one of the first large scale benchmark study of using machine learning for crop loss classification at field parcel scale. The classification setting and trained models have numerous potential applications, for example, allowing government agencies or insurance companies to verify crop-loss claims by farmers and realise efficient agricultural monitoring.

Imputing pre-diagnosis health behaviour in cancer registry data and investigating its relationship with oesophageal cancer survival time.

As oesophageal cancer has short survival, it is likely pre-diagnosis health behaviours will have carry-over effects on post-diagnosis survival times. Cancer registry data sets do not usually contain pre-diagnosis health behaviours and so need to be augmented with data from external health surveys. A new algorithm is introduced and tested to augment cancer registries with external data when one-to-one data linkage is not available. The algorithm is to use external health survey data to impute pre-diagnosis health behaviour for cancer patients, estimate misclassification errors in these imputed values and then fit misclassification corrected Cox regression to quantify the association between pre-diagnosis health behaviour and post-diagnosis survival. Data from US cancer registries and a US national health survey are used in testing the algorithm. It is demonstrated that the algorithm works effectively on simulated smoking data when there is no age confounding. But age confounding does exist (risk of death increases with age and most health behaviours change with age) and interferes with the performance of the algorithm. The estimate of the hazard ratio (HR) of pre-diagnosis smoking was HR = 1.32 (95% CI 0.82,2.68) with HR = 1.93 (95% CI 1.08,7.07) in the squamous cell sub-group and pre-diagnosis physical activity was protective of survival with HR = 0.25 (95% CI 0.03, 0.81). But the method failed for less common behaviours (such as heavy drinking). Further improvements in the I2C2 algorithm will permit enrichment of cancer registry data through imputation of new variables with negligible risk to patient confidentiality, opening new research opportunities in cancer epidemiology.

Choosing Imputation Models

AbstractImputing missing values is an important preprocessing step in data analysis, but the literature offers little guidance on how to choose between imputation models. This letter suggests adopting the imputation model that generates a density of imputed values most similar to those of the observed values for an incomplete variable after balancing all other covariates. We recommend stable balancing weights as a practical approach to balance covariates whose distribution is expected to differ if the values are not missing completely at random. After balancing, discrepancy statistics can be used to compare the density of imputed and observed values. We illustrate the application of the suggested approach using simulated and real-world survey data from the American National Election Study, comparing popular imputation approaches including random forests, hot-deck, predictive mean matching, and multivariate normal imputation. An R package implementing the suggested approach accompanies this letter.

Adaptive Deep Incremental Learning — Assisted Missing Data Imputation for Streaming Data

With the rapid explosion of the data streams from the applications, ensuring accurate data analysis is essential for effective real-time decision making. Nowadays, data stream applications often confront the missing values that affect the performance of the classification models. Several imputation models have adopted the deep learning algorithms for estimating the missing values; however, the lack of parameter and structure tuning in classification, degrade the performance for data imputation. This work presents the missing data imputation model using the adaptive deep incremental learning algorithm for streaming applications. The proposed approach incorporates two main processes: enhancing the deep incremental learning algorithm and enhancing deep incremental learning-based imputation. Initially, the proposed approach focuses on tuning the learning rate with both the Adaptive Moment Estimation (Adam) along with Stochastic Gradient Descent (SGD) optimizers and tuning the hidden neurons. Secondly, the proposed approach applies the enhanced deep incremental learning algorithm to estimate the imputed values in two steps: (i) imputation process to predict the missing values based on the temporal-proximity and (ii) generation of complete IoT dataset by imputing the missing values from both the predicted values. The experimental outcomes illustrate that the proposed imputation model effectively transforms the incomplete dataset into a complete dataset with minimal error.