Hot Deck Imputation Research Articles

An integrated framework for rainfall prediction and analysis using a Stacked Heterogeneous Ensemble Model (SHEM)

Recent climate conditions in India have led to numerous disasters, including floods, overflowing rivers, broken dams, and reduced vegetation. Machine learning aims to train models to make predictions based on historical data. Although many studies have focused on predictions and model building, this research introduces a novel integrated framework using a Stacked Heterogeneous Ensemble Model (SHEM). It aims to enhance prediction accuracy by employing another novel approach for imputing missing values, the Variable Specific Hot Deck (VSHD) imputation method. The outcomes were contrasted with established machine learning techniques, including random forest, decision tree, k-nearest neighbor, and support vector machine. After completing the imputation, we proceeded to implement the SHEM model. Real-time climate data for the Cuddalore location in Tamil Nadu state, South India was collected from the NASA Power Access viewer portal to verify the accuracy level of the proposed model. Performance analysis indicates that the proposed imputation outperforms all four alternative models with a 30% average improvement in accuracy. Moreover, the developed SHEM model has a reduced RMSE value of 0.321 and an R-squared value of 0.952 and shows a 9% improvement in accuracy compared to the base model’s performance. Furthermore, these prediction results are compared to an LSTM (Long Short Term Memory), a deep learning model to calculate accuracy and loss, showing that the proposed achieves high accuracy and significant loss during validation. The obtained results can serve as a guideline for atmospheric scientists and various weather forecasting applications, helping them to choose the most appropriate machine learning method for their prediction task.

Exogenous factors associated with legacy PFAS concentrations in the general U.S. population: NHANES 1999-2018

To reduce per- and polyfluoroalkyl substances (PFAS) exposure through legislation and other interventions, we must understand factors contributing to individual body burden. Identifying factors associated with serum perfluorinated compounds (PFAS) concentrations also aids in identifying groups at higher risk of adverse health outcomes due to elevated exposure. This exploratory analysis provides initial findings on exposure-related factors associated with legacy PFAS concentrations in the general United States (U.S.) population. We obtained National Health and Nutrition Examination Survey (NHANES) datasets with individual serum PFAS measurements from cycle years 1999-2018 (N = 14,961), excluding the 2001-2002 cycle due to pooled sampling. Over 100 features were evaluated for associations with PFAS concentrations. Data were singly imputed using hot deck and predictive mean matching (PMM) methods, and model performance by imputation method was compared using elastic net regression (ENR) models. Hot-deck imputation explained the most variance in predictive models; thus, a dataset imputed via hot-deck was used for feature selection. ENR models were employed to identify the top variables associated with legacy PFAS concentrations, and selected features were put into linear mixed models to obtain beta estimates and standard errors. Survey year, demographic, income, place of birth and citizenship status, household size, dietary, health, food insecurity, general health and healthcare, housing, social and behavioral, and other characteristics were important factors associated with legacy PFAS concentrations in this nationally representative study population. A better understanding of exogenous factors associated with PFAS concentrations can influence future epidemiological studies, guiding decisions on adjustment for confounding, and advancing our understanding of factors that affect chemical half-lives and toxicokinetics.

Use of Sequential Hot-Deck Imputation for Missing Health Care Systems Data for Population Health Research.

Electronic medical record (EMR) data present many opportunities for population health research. The use of EMR data for population risk models can be impeded by the high proportion of missingness in key patient variables. Common approaches like complete case analysis and multiple imputation may not be appropriate for some population health initiatives that require a single, complete analytic data set. In this study, we demonstrate a sequential hot-deck imputation (HDI) procedure to address missingness in a set of cardiometabolic measures in an EMR data set. We assessed the performance of sequential HDI within the individual variables and a commonly used composite risk score. A data set of cardiometabolic measures based on EMR data from 2 large urban hospitals was used to create a benchmark data set with simulated missingness. Sequential HDI was applied, and the resulting data were used to calculate atherosclerotic cardiovascular disease risk scores. The performance of the imputation approach was assessed using a set of metrics to evaluate the distribution and validity of the imputed data. Of the 567,841 patients, 65% had at least 1 missing cardiometabolic measure. Sequential HDI resulted in the distribution of variables and risk scores that reflected those in the simulated data while retaining correlation. When stratified by age and sex, risk scores were plausible and captured patterns expected in the general population. The use of sequential HDI was shown to be a suitable approach to multivariate missingness in EMR data. Sequential HDI could benefit population health research by providing a straightforward, computationally nonintensive approach to missing EMR data that results in a single analytic data set.

Improving Donor Imputation Using the Prediction Power of Random Forests: a Combination of SwissCheese and missForest

Abstract Imputation procedures are frequently used to treat nonresponse. With random hot deck imputation, missing values are replaced by valid observed values from other units in the same dataset. The recently developed balanced nearest neighbor imputation method, implemented in the SwissCheese R package, generates random hot deck imputation under certain balancing constraints to decrease the variance of the total estimator, in the presence of multivariate nonresponse. The method relies on a notion of neighborhood between units, utilizing a distance measure that becomes difficult to define in high dimensions. In contrast to hot deck imputation methods, many imputation procedures obtain replacement values from prediction models fit from observed data. The missForest method, which uses random forests as prediction models, is an example of this approach. In this article, we propose a new approach that uses the two methods in a complementary manner. We refine the distance measure in the SwissCheese method using missForest predictions. Through a simulation study on empirical data from the Swiss Survey on Income and Living Conditions, we demonstrate reductions in Monte Carlo variance, bias, and mean squared error of the totals obtained by our proposed imputed estimator compared to those obtained using SwissCheese alone.

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

Parallel fractional hot-deck imputation (P-FHDI [1]) is a general-purpose, assumption-free tool for handling item nonresponse in big incomplete data by combining the theory of FHDI and parallel computing. FHDI cures multivariate missing data by filling each missing unit with multiple observed values (thus, hot-deck) without resorting to distributional assumptions. P-FHDI can tackle big incomplete data with millions of instances (big- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> ) or 10,000 variables (big- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> ). However, handling ultra incomplete data (i.e., concurrently big- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> and big- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> ) with tremendous instances and high dimensionality has posed problems to P-FHDI due to excessive memory requirement and execution time. To tackle the aforementioned challenges, we propose the ultra data-oriented P-FHDI (named UP-FHDI) capable of curing ultra incomplete data. In addition to the parallel Jackknife method, this paper enables a computationally efficient ultra data-oriented variance estimation using parallel linearization techniques. Results confirm that UP-FHDI can tackle an ultra dataset with one million instances and 10,000 variables. This paper illustrates the special parallel algorithms of UP-FHDI and confirms its positive impact on the subsequent deep learning performance.

Missing data approaches in longitudinal studies of aging: A case example using the National Health and Aging Trends Study.

Missing data is a key methodological consideration in longitudinal studies of aging. We described missing data challenges and potential methodological solutions using a case example describing five-year frailty state transitions in a cohort of older adults. We used longitudinal data from the National Health and Aging Trends Study, a nationally-representative cohort of Medicare beneficiaries. We assessed the five components of the Fried frailty phenotype and classified frailty based on their number of components (robust: 0, prefrail: 1-2, frail: 3-5). One-, two-, and five-year frailty state transitions were defined as movements between frailty states or death. Missing frailty components were imputed using hot deck imputation. Inverse probability weights were used to account for potentially informative loss-to-follow-up. We conducted scenario analyses to test a range of assumptions related to missing data. Missing data were common for frailty components measured using physical assessments (walking speed, grip strength). At five years, 36% of individuals were lost-to-follow-up, differentially with respect to baseline frailty status. Assumptions for missing data mechanisms impacted inference regarding individuals improving or worsening in frailty. Missing data and loss-to-follow-up are common in longitudinal studies of aging. Robust epidemiologic methods can improve the rigor and interpretability of aging-related research.

Imputation-based Q-learning for optimizing dynamic treatment regimes with right-censored survival outcome.

Q-learning has been one of the most commonly used methods for optimizing dynamic treatment regimes (DTRs) in multistage decision-making. Right-censored survival outcome poses a significant challenge to Q-Learning due to its reliance on parametric models for counterfactual estimation which are subject to misspecification and sensitive to missing covariates. In this paper, we propose an imputation-based Q-learning (IQ-learning) where flexible nonparametric or semiparametric models are employed to estimate optimal treatment rules for each stage and then weighted hot-deck multiple imputation (MI) and direct-draw MI are used to predict optimal potential survival times. Missing data are handled using inverse probability weighting and MI, and the nonrandom treatment assignment among the observed is accounted for using a propensity-score approach. We investigate the performance of IQ-learning via extensive simulations and show that it is more robust to model misspecification than existing Q-Learning methods, imputes only plausible potential survival times contrary to parametric models and provides more flexibility in terms of baseline hazard shape. Using IQ-learning, we developed an optimal DTR for leukemia treatment based on a randomized trial with observational follow-up that motivated thisstudy.

Missing value imputation Techniques: A Survey

Numerous of information is being accumulated and placed away every day. Big quantity of misplaced areas in a dataset might be a large problem confronted through analysts due to the fact it could cause numerous issues in quantitative investigates. To handle such misplaced values, numerous methods were proposed. This paper offers a review on different techniques available for imputation of unknown information, such as median imputation, hot (cold) deck imputation, regression imputation, expectation maximization, help vector device imputation, multivariate imputation using chained equation, SICE method, reinforcement programming, non-parametric iterative imputation algorithms, and multilayer perceptrons. This paper also explores a few satisfactory choices of methods to estimate missing values to be used by different researchers on this discipline of study. Furthermore, it aims to assist them to discern out what approach is commonly used now, the overview may additionally provide a view of every technique alongside its blessings and limitations to take into consideration of future studies on this area of study. It can be taking into account as baseline to solutions the question which techniques were used and that is the maximum popular.

The Impact of Missing Data on the Performances of DIF Detection Methods

This study analyzed the impact of missing data techniques on performances of two differential item functioning (DIF) detection methods (Mantel Haenszel and Multiple Indicator and Multiple Causes) under missing completely at random missing data mechanism. Percentage of missing data was set at 5% and 15%. Zero imputation, listwise deletion and fractional hot-deck imputation were used to handle missing data. The data set of the study consisted of 17 items in the S12 item cluster of Programme for International Student Assessment (PISA) 2015 science test. Results showed that fractional hot-deck imputation produced the best results in identifying DIF items in all conditions and it had also the closest DIF values to the values obtained from complete data set. It was also found that multiple indicator and multiple causes method was more adversely affected than Mantel Haenszel by the presence of missing data.

범주형 전환문항을 포함하는 항목 무응답 대체 방법 비교

In social surveys, unfolding bracket questions are used as a technique to reduce the loss of information due to nonresponse when item nonresponses occur. Item nonresponse, including unfolding bracket questions, has a similar structure to interval censored survival data. However, imputation methods to handle item nonresponses including unfolding bracket questions are limited. In this study, we examine performance of imputation with and without unfolding bracket questions. Also we compare three imputation methods to handle unfolding bracket questions. The First method is a linear regression-based nearest neighbor hotdeck imputation. Secondly, it is possible to consider a uniform distribution that selects a value with the same probability within the section for each individual. The third is an imputation method using the survival function estimated by the nonparametric maximum likelihood estimation method. Hotdeck imputation was considered as an imputation for not using unfolding bracket questions. The performance of these imputation methods was evaluated through simulation and the KLoSA study is used to provide examples of the application of these methods to real data. Imputation methods using unfolding bracket questions showed better performance than the method without unfolding bracket questions. The performance of the imputation method using the survival function estimated by the nonparametric maximum likelihood estimation method was better when the nonresponse rate was low and when the nonresponse rate is large in terms of root mean square error. When the non-response rate is large, the linear regression-based nearest neighbor hotdeck imputation method showed better results in terms of mean bias.

Empirical Comparison of Imputation Methods for Multivariate Missing Data in Public Health.

Sample estimates derived from data with missing values may be unreliable and may negatively impact the inferences that researchers make about the underlying population due to nonresponse bias. As a result, imputation is often preferred to listwise deletion in handling multivariate missing data. In this study, we compared three popular imputation methods: sequential multiple imputation, fractional hot-deck imputation, and generalized efficient regression-based imputation with latent processes for handling multivariate missingness under different missing patterns by conducting descriptive and regression analyses on the imputed data and seeing how the estimates differ from those generated from the full sample. Limited Monte Carlo simulation results by using the National Health Nutrition and Examination Survey and Behavioral Risk Factor Surveillance System are presented to demonstrate the effect of each imputation method on reducing bias and increasing efficiency for the parameter estimate of interest for that particular incomplete variable. Although these three methods did not always outperform listwise deletion in our simulated missing patterns, they improved many descriptive and regression estimates when used to impute all incomplete variables at once.

A comparison of imputation methods for categorical data

ObjectivesMissing data is commonplace in clinical databases, which are being increasingly used for research. Without giving any regard to missing data, results from analysis may become biased and unrepresentative. Clinical databases contain mainly categorical variables. This study aims to assess the methods used for imputation in categorical variables. Materials and methodsWe utilized data extracted from paper-based maternal health records from Kawempe National Referral Hospital, Uganda. We compared the following imputation methods for categorical data in an empirical analysis: Mode, K-Nearest Neighbors (KNN), Random Forest (RF), Sequential Hot-Deck (SHD), and Multiple Imputation by Chained Equations (MICE). The five imputation methods were first compared by accuracy of predicting the missing values. Next, the imputation methods were compared by predictive accuracy of the outcome variable in four classifiers. The consistency of performance of imputation methods across different levels of missing data (5%–50 %) was assessed by Kendall's W test. ResultsKNN imputation had the highest precision score at levels (5%–50 %) of MCAR missing data. At lower proportions of missing data (5 %, 10 %, 15 %, 20 %), RF imputation had the second-highest precision score. SHD imputation had the worst precision at all levels of missing data. In the prediction of the outcome, the methods performed differently at all proportions of missing data in the four classifiers. Even though KNN imputation was the best method in predicting the missing values, it did not consistently enhance the predictive accuracy of the classifiers at all levels of missing data. Our findings show that a high precision score of an imputation method does not translate into higher predictive accuracy in classifiers. ConclusionsKNN imputation is the best method in predicting missing values in categorical variables. There is no universal best imputation method that yields the highest predictive accuracy at all proportions of missing data.

Implementing Multiple Imputation for Missing Data in Longitudinal Studies When Models are Not Feasible: An Example Using the Random Hot Deck Approach.

Researchers often use model-based multiple imputation to handle missing at random data to minimize bias. However, constraints within the data may sometimes result in implausible values, making model-based imputation infeasible. In these contexts, we illustrate how random hot deck imputation can allow for plausible multiple imputation in longitudinal studies. Our motivating example is the Childhood Health, Activity, and Motor Performance School Study Denmark (CHAMPS-DK), a prospective cohort study that measured weekly sports participation for 1700 Danish schoolchildren. Using observed data on 4 variables (pain, activity frequency, sport, sport counts), we created a gold-standard data set without missing data. We then created a synthetic data set by setting some variable values to missing based on a prediction model that mimicked real-data missingness patterns. To create 5 imputed data sets, we matched each record with missing data to several fully observed records, generated probabilities from matched records, and sampled from these records based on the probability of each occurring. We assessed variability and agreement (kappa) between the imputed data sets and the gold-standard data set. We compare results to common model-based imputation methods. Variability across data sets appeared reasonable. The range of kappa for the random hot deck approach was moderate for activity frequency (0.65 to 0.71) and sport (0.59 to 0.85), and poor for common model-based approaches (range 0.00 to 0.11). The range of kappas for sport count was strong (0.87 to 0.97) for random hot deck imputation and weak to moderate (0.55 to 0.71) for common model-based imputation. Agreement was higher when more information was present, and when prevalence was higher for our binary variable sport. Random hot deck imputation should be considered as an alternative method when model-based approaches are infeasible, specifically where there are constraints within and between covariates.

Item non-response imputation in the Korea National Health and Nutrition Examination Survey.

The Korea National Health and Nutrition Examination Survey (KNHANES) is a public health survey that assesses individuals' health and nutritional status and monitors the prevalence of major chronic diseases. In general, sampling weights are adjusted for unit non-responses and imputation is conducted for item non-responses. In this study, we proposed strategies for imputing item non-responses in the KNHANES in order to improve the usefulness of data, minimize bias, and increase statistical power. After applying logical imputation, we adopted 2 separate imputation methods for each variable type: unweighted sequential hot-deck imputation for categorical variables and sequential regression imputation for continuous variables. For variance estimation, multiple imputations were applied to the continuous variables. To evaluate the performance of the proposed strategies, we compared the marginal distributions of imputed variables and the results of multivariable regression analysis for the complete-case data and the expanded data with imputed values, respectively. When comparing the marginal distributions, most non-responses were imputed. The multivariable regression coefficients presented similar estimates; however, the standard errors decreased, resulting in statistically significant p-values. The proposed imputation strategies may cope with the loss of precision due to missing data, thus enhancing statistical power in analyses of the KNHANES by providing expanded data with imputed values. The proposed imputation strategy may enhance the utility of data by increasing the number of complete cases and reducing the bias in the analysis, thus laying a foundation to cope with the occurrence of item non-responses in further surveys.

Comparison of Seventeen Missing Value Imputation Techniques

Copious data are collected and put away each day. That information can be utilized to extricate curiously designs. However, the information that we collect is ordinarily inadequate. Presently, utilizing that information to extricate any data may allow deceiving comes about. Utilizing that, we pre-process the information to exterminate the variations from the norm. In case of a low rate of lost values, those occurrences can be overlooked, but, in the case of huge sums, overlooking them will not allow wanted results. Many lost spaces in a dataset could be a huge issue confronted by analysts because it can lead to numerous issues in quantitative investigations. So, performing any information mining procedures to extricate a little good data out of a dataset, a few pre-processings of information can be done to dodge such paradoxes and, in this manner, move forward the quality of information. For handling such lost values, numerous methods have been proposed since 1980. The best procedure is to disregard the records containing lost values. Another method is ascription, which includes supplanting those lost spaces with a few gauges by doing certain computations. This would increment the quality of information and would extemporize forecast comes about. This paper gives an audit on methods for handling lost information like median imputation (MDI), hot (cold) deck imputation, regression imputation, expectation maximization (EM), support vector machine imputation (SVMI), multivariate imputation by chained equation (MICE), SICE technique, reinforcement programming, nonparametric iterative imputation algorithms (NIIA), and multilayer perceptrons. This paper also explores some good options of methods to estimate missing values to be used by other researchers in this field of study. Also, it aims to help them to figure out what method is commonly used now. The overview may also provide insight into each method and its advantages and limitations to consider for future research in this field of study. It can be a baseline to answer the questions of which techniques have been used and which is the most popular.

Predicting initiation to treatment for youth on probation: A multi-level approach

BackgroundPrevious research on youth involved in the juvenile legal system (JLS), who have higher substance use prevalence than other adolescents, has focused on treatment outcomes post-initiation. Scant research has examined factors affecting treatment initiation following JLS referral. The current work addresses this gap by estimating net impacts of state, site, and client attributes on initiating treatment. MethodsMixed effects logistic regression models analyzed data from the Behavioral Health Services Cascade framework of the JJ-TRIALS project for 5325 youth records from 22 sites across six states, using weighted hot deck imputation to address missing data. The analysis permitted estimating net impacts of individual factors while controlling for state variation factors and separating out within-state site variation. ResultsRates of treatment initiation varied significantly across sites. About 63% of outcome variation arose from site and state disparities. The odds of initiating treatment were 127% higher for youth under a higher supervision level compared to youth under low supervision. Race and ethnicity were significantly associated with initiation, with Black and Hispanic youth having 44% and 42% lower odds of initiating treatment, respectively. ConclusionIndividual factors, including race, ethnicity, and level of supervision, significantly differed in treatment initiation, controlling for between-state variation and allowing between-site-within-state effects to operate separately. A deeper understanding of factors shaping initiation may help behavioral health and JLS agencies to develop strategies to increase initiation for referred youth. This is the first study delineating individual, cross-program, and between-state variation in treatment initiation among referred youth.

Retrospective analysis to describe trends in first-ever prostate-specific antigen (PSA) testing for primary healthcare facilities in the Gauteng Province, South Africa, between 2006 and 2016

ObjectivesThe objective of our study was to use laboratory data to describe prostate-specific antigen (PSA) testing trends for primary healthcare (PHC) services from a single province. PHC is a basic...

A Rescaling Bootstrap Approach For Imputed Survey Data

Abstract Imputation is usually used to deal with item nonresponse in surveys. Treating the imputed values as true observations may obviously lead to serious underestimation of the variance of point estimators. In this article, we propose a new bootstrap method under the rescaling bootstrap approach for estimating the variance of an imputed estimator obtained after applying deterministic regression or random hot-deck imputation. A novel technique is used to rescale the original data set through solving certain systems of linear equations. The proposed procedure can handle unequal response probabilities and large sampling fractions. Some simulation studies are conducted to show the great performance of the proposed method in terms of relative bias, relative efficiency, and coverage probability, for both population mean and median.

A note on imputing squares via polynomial combination approach

The polynomial combination (PC) method, proposed by Vink and Van Buuren, is a hot-deck multiple imputation method for imputation models containing squared terms. The method yields unbiased regression estimates and preserves the quadratic relationships in the imputed data for both MCAR and MAR mechanisms. However, Vink and Van Buuren never studied the coverage rate of the PC method. This paper investigates the coverage of the nominal 95% confidence intervals for the polynomial combination method and improves the algorithm to avoid the perfect prediction issue. We also compare the original and the improved PC method to the substantive model compatible fully conditional specification method proposed by Bartlett et al. and elucidate the two imputation methods’ characters.

Adapting Nearest Neighbor for Multiple Imputation: Advantages, Challenges, and Drawbacks

Abstract The U.S. Census Bureau has historically used nearest neighbor (NN) or random hot deck (RHD) imputation to handle missing data for many types of survey data. Using these methods removes the need to parametrically model values in imputation models. With strong auxiliary information, NN imputation is preferred because it produces more precise estimates than RHD. In addition, NN imputation is robust against a misspecified response mechanism if missingness depends on the auxiliary variable, in contrast to RHD which ignores the auxiliary information. A compromise between these two methods is k-NN imputation, which identifies a set of the k closest neighbors (“donor pool”) and randomly selects a single donor from this set. Recently these methods have been used for multiple imputation (MI), enabling variance estimation via the so-called Rubin’s Combining Rules. The Approximate Bayesian Bootstrap (ABB) is a simple-to-implement algorithm that makes the RHD “proper” for MI. In concept, ABB should work to propagate uncertainty for NN MI; bootstrapping respondents mean each nonrespondent’s one “nearest” donor will not be available for every imputation. However, we demonstrate through simulation that NN MI using ABB leads to variance underestimation. This underestimation is somewhat but not entirely attenuated with k-NN imputation. An alternative approach to variance estimation after MI, bootstrapped MI, eliminates the underestimation with NN imputation, but we show that it suffers from overestimation of variance with nonnegligible sampling fractions under both equal and unequal probability sampling designs. We propose a modification to bootstrapped MI to account for nonnegligible sampling fractions. We compare the performance of RHD and the various NN MI methods under a variety of sampling designs, sampling fractions, distribution shapes, and missingness mechanisms.