Missing Data Structure Research Articles

The Effect of Regularized Regression and Tree-Based Missing Data Imputation Methods on Classification Performance in High Dimensional Data

Missing data is an important problem in the analysis and classification of high dimensional data. The aim of this study is to compare the effects of four different missing data imputation methods on classification performance in high dimensional data. In this study, missing data imputation methods were evaluated using data sets, whose independent variables between mixed correlated with each other, for binary dependent variable, p=500 independent variables, n=150 units and 1000 times running simulation. Missing data structures were created according to different missing rates. Different datasets were obtained by imputing the missing values using different methods. Regularized regression methods such as lasso and elastic net regression were used for imputation, as well as tree-based methods such as support vector machine and classification and regression trees. At the end of simulation, the classification scores of the methods were obtained by gradient boosting machines and the missing data prediction performances were evaluated according to the distance of these scores from the reference. Our simulation demonstrates that regularized regression methods outperform tree-based methods in classifying high dimensional datasets. Additionally, it was found that the increase in the amount of missing values reduced the classification performance of the methods in high dimensional data.

Assessment of Different Methods for Estimation of Missing Rainfall Data

Missing data is a common problem encountered in various fields, including clinical research, environmental sciences and hydrology. In order to obtain reliable results from the analysis, the data inventory must be completed. This paper presents a methodology for addressing the missing data problem by examining the missing data structure and missing data techniques. Simulated datasets were created by considering the number of missing data, missing data pattern and missing data mechanism of real datasets containing missing values, which are often overlooked in hydrology. Considering the missing data pattern, the most commonly used methods for missing data analysis in hydrology and other fields were applied to the created simulated datasets. Simple imputation techniques and expectation maximization (EM) were implemented in SPSS software and machine learning techniques such as k-nearest neighbor (kNN), together with the hot-deck were implemented in the Python programming language. In the performance evaluation based on error metrics, it is concluded that the EM method is the most suitable completion method. Homogeneity analyses were performed in the Mathematica programming language to identify possible changes and inconsistencies in the completed rainfall dataset. Homogeneity analyses revealed that most of the completed rainfall datasets are homogeneous at class 1 level, consistent and reliable and do not show systematic changes in time.

Functional principal component analysis for incomplete space–time data

Environmental signals, acquired, e.g., by remote sensing, often present large gaps of missing observations in space and time. In this work, we present an innovative approach to identify the main variability patterns, in space–time data, when data may be affected by complex missing data structures. We formalize the problem in the framework of functional data analysis, proposing an innovative method of functional principal component analysis (fPCA) for incomplete space–time data. The functional nature of the proposed method permits to borrow information from measurements observed at nearby spatio-temporal locations. The resulting functional principal components are smooth fields over the considered spatio-temporal domain, and can lead to interesting insights in the spatio-temporal dynamic of the phenomenon under study. Moreover, they can be used to provide a reconstruction of the missing entries, also under severe missing data patterns. The proposed model combines a weighted rank-one approximation of the data matrix with a roughness penalty. We show that the estimation problem can be solved using a majorize–minimization approach, and provide a numerically efficient algorithm for its solution. Thanks to a discretization based on finite elements in space and B-splines in time, the proposed method can handle multidimensional spatial domains with complex shapes, such as water bodies with complicated shorelines, or curved spatial regions with complex orography. As shown by simulation studies, the proposed space–time fPCA is superior to alternative techniques for Principal Component Analysis with missing data. We further highlight the potentiality of the proposed method for environmental problems, by applying space–time fPCA to the study of the lake water surface temperature (LWST) of Lake Victoria, in Central Africa, starting from satellite measurements with large gaps. LWST is considered one of the fundamental indicators of how climate change is affecting the environment, and is recognized as an essential climate variable.

Comparing the performance of eight imputation methods for propensity score matching in missing data problem

Propensity score (PS) is a popular method to control for covariates in observational studies. A challenge in PS analyses is missing values in covariates. This study aims to investigate how different imputation methods of handling missing values of covariates in a PS analysis can affect average treatment on the treated (ATT) estimates. In this study, missing data imputation methods were evaluated using different data sets, whose covariates were low, medium, and high (r=0.10, 0.50, 0.85) correlated with each other, for n=200 units and 1000 times running simulation. Missing data structures were created according to the missing at random (MAR) mechanism and different missing rates. Different datasets were obtained after having imputed the missing values separately by eight imputation methods including mean, median, mode, hot deck, last observation carried forward (LOCF), next observation carried backward (NOCB), regression and predictive mean matching (PMM). Then the PS nearest neighbor matching was implemented and ATT scores were obtained using the imputed data sets. The predictive performance of imputation methods was compared according to ATT scores by hierarchical cluster analysis with Euclidean distance complete linkage. ATT scores of regression and PMM methods were closer to each other and these methods showed the best predictive performance. Additionally, when there were larger amounts of missing data, the PMM was the best method of choice. Ignoring missing values on covariates for PS analyses causes information loss significantly and this information loss becomes greater as the rate of missing data increases. PS analyses might be biased if missing data on covariates are also ignored. To prevent this information loss and bias, PS analyses should be performed after solving the problem of missing data with MAR mechanism on covariates by regression and PMM methods, which showed statistical superiority compared to other methods in this study.

Adaptive personalized prior-knowledge-informed model predictive control for type 1 diabetes

This work considers the problem of adaptive prior-informed model predictive control (MPC) formulations that explicitly incorporate prior knowledge in the model development and is robust to missing data in the output measurements. The proposed prediction model is based on a latent variables model to extract glycemic dynamics from highly-correlated data and incorporates prior knowledge of exponential stability to improve the prediction ability. Missing data structures are formulated to enable model predictions when output measurements are missing for short periods of time. Based on the latent variables model, the MPC strategy and adaptive rules are developed to automatically tune the aggressiveness of the MPC. The adaptive prior-knowledge-informed MPC is evaluated with computer simulations for the control of blood glucose concentrations in people with Type 1 diabetes (T1D) using simulated virtual patients. Due to the variability among people with T1D, the hyperparameters of the prior-knowledge-informed model are personalized to individual subjects. The percentage of time spent in the target range is 76.48% when there are no missing data and 76.52% when there are missing data episodes lasting up to 30mins (6 samples). Incorporating the adaptive rules further improves the percentage of time in target range to 84.58% and 84.88% for cases with no missing data and missing data, respectively. The proposed adaptive prior-informed MPC formulation provides robust, effective, and safe regulation of glucose concentration in T1D despite disturbances and missing measurements.

Bayesian finite-population inference with spatially correlated measurements

Community-based public health interventions often rely on representative, spatially referenced outcome data to draw conclusions about a finite population. To estimate finite-population parameters, we are posed with two challenges: to correctly account for spatial association among the sampled and nonsampled participants and to correctly model missingness in key covariates, which may be also spatially associated. To accomplish this, we take inspiration from the preferential sampling literature and develop a general Bayesian framework that can specifically account for preferential non-response. This framework is first applied to three missing data scenarios in a simulation study. It is then used to account for missing data patterns seen in reported annual household income in a corner-store intervention project. Through this, we are able to construct finite-population estimates of the percent of income spent on fruits and vegetables. Such a framework provides a flexible way to account for spatial association and complex missing data structures in finite populations.

Bayesian network-based missing mechanism identification (BN-MMI) method in medical research

BackgroundTraditional approaches to identify missing mechanisms are usually based on the hypothesis test and confronted with both theoretical and practical challenges. It has been proved that the Bayesian network is powerful in integrating, analyzing and visualizing information, and some previous researches have verified the promising features of Bayesian network to deal with the aforementioned challenges in missing mechanism identification. Based on the above reasons, this paper explores the application of Bayesian network to the identification of missing mechanisms for the first time, and proposes a new method, the Bayesian network-based missing mechanism identification (BN-MMI) method, to identify missing mechanism in medical research.MethodsThe procedure of BN-MMI method consists three easy-to-implement steps: estimating the missing data structure by the Bayesian network; assessing the credibility of the estimated missing data structure; and identifying the missing mechanism from the estimated missing data structure. The BN-MMI method is verified by simulation research and empirical research.ResultsThe simulation study verified the validity, consistency and robustness of BN-MMI method, and indicated its outperformance in contrast to the traditional logistic regression method. In addition, the empirical study illustrated the applicability of BN-MMI method in the real world by an example of medical record data.ConclusionsIt was confirmed that the BN-MMI method itself, together with human knowledge and expertise, could identify the missing mechanisms according to the probabilistic dependence/independence relations among variables of interest. At the same time, our research shed light upon the potential application of BN-MMI method to a broader range of missing data issues in medical studies.

SPEARS: Standard Performance Evaluation of Ancestral haplotype Reconstruction through Simulation.

MotivationAncestral haplotype maps provide useful information about genomic variation and insights into biological processes. Reconstructing the descendent haplotype structure of homologous chromosomes, particularly for large numbers of individuals, can help with characterizing the recombination landscape, elucidating genotype-to-phenotype relationships, improving genomic predictions and more. Inferring haplotype maps from sparse genotype data is an efficient approach to whole-genome haplotyping, but this is a non-trivial problem. A standardized approach is needed to validate whether haplotype reconstruction software, conceived population designs and existing data for a given population provides accurate haplotype information for further inference.ResultsWe introduce SPEARS, a pipeline for the simulation-based appraisal of genome-wide haplotype maps constructed from sparse genotype data. Using a specified pedigree, the pipeline generates virtual genotypes (known data) with genotyping errors and missing data structure. It then proceeds to mimic analysis in practice, capturing sources of error due to genotyping, imputation and haplotype inference. Standard metrics allow researchers to assess different population designs and which features of haplotype structure or regions of the genome are sufficiently accurate for analysis. Haplotype maps for 1000 outcross progeny from a multi-parent population of maize are used to demonstrate SPEARS.Availabilityand implementationSPEARS, the protocol and suite of scripts, are publicly available under an MIT license at GitHub (https://github.com/maizeatlas/spears).Supplementary information Supplementary data are available at Bioinformatics online.

BEST: a decision tree algorithm that handles missing values

The main contribution of this paper is the development of a new decision tree algorithm. The proposed approach allows users to guide the algorithm through the data partitioning process. We believe this feature has many applications but in this paper we demonstrate how to utilize this algorithm to analyse data sets containing missing values. We tested our algorithm against simulated data sets with various missing data structures and a real data set. The results demonstrate that this new classification procedure efficiently handles missing values and produces results that are slightly more accurate and more interpretable than most common procedures without any imputations or pre-processing.

Investigating Measurement Invariance under Different Missing Value Reduction Methods

This study aims to comparatively examine the resultant findings by testing the measurement invariance with structural equation modeling in cases where the missing data is handled using the expectation-maximization (EM), regression imputation, and mean substitution methods in the complete data matrix and the 5% missing data matrix that is randomly obtained from the same matrix. The data were collected from 2822 students. Of these students, who participated in the study, 1338 (49.2%) were females while 1434 (50.8%) were males. The data were collected using the “School Attitude Scale” developed by Alıcı (2013). In this study, the measurement invariance was tested with structural equation modeling in the complete data matrix and in cases of handling the missing data it was tested using EM, Regression-Based Imputation, and Mean Substitution methods. In the present study the measurement invariance decisions taken for the complete data matrix coincide with the mean substitution method in all sub-factors, with regression imputation from the other suggested methods in two sub-factors, and with expectation-maximization in one sub-factor. It was concluded that different data imputation methods change measurement invariance decisions, considering all the factors. The three techniques provided consistent results only in one sub-dimension. The ways of dealing with the missing data change the results; thereby, increasing fundamental studies in this regard is necessary. With reference to these results, it would be worthwhile to study different missing data structures and different proportions of missing data in terms of invariance decisions.

Traffic Data Imputation and Prediction: An Efficient Realization of Deep Learning

In this paper, we study the prediction of traffic flow in the presence of missing information from data set. Specifically, we adopt three different patterns to model the missing data structure, and apply two types of approaches for the imputation. In consequence, a forecasting model via deep learning based methods is proposed to predict the traffic flow from the recovered data set. The experiments demonstrate the effectiveness of using deep learning based imputation in improving the accuracy of traffic flow prediction. Based on the experimental results, we conduct a thorough discussion on the appropriate methods to predict traffic flow under various missing data conditions, and thus shedding the light for a practical design.

A strategy for high-dimensional multivariable analysis classifies childhood asthma phenotypes from genetic, immunological, and environmental factors.

BackgroundAssociations between childhood asthma phenotypes and genetic, immunological, and environmental factors have been previously established. Yet, strategies to integrate high‐dimensional risk factors from multiple distinct data sets, and thereby increase the statistical power of analyses, have been hampered by a preponderance of missing data and lack of methods to accommodate them.MethodsWe assembled questionnaire, diagnostic, genotype, microarray, RT‐qPCR, flow cytometry, and cytokine data (referred to as data modalities) to use as input factors for a classifier that could distinguish healthy children, mild‐to‐moderate allergic asthmatics, and nonallergic asthmatics. Based on data from 260 German children aged 4‐14 from our university outpatient clinic, we built a novel multilevel prediction approach for asthma outcome which could deal with a present complex missing data structure.ResultsThe optimal learning method was boosting based on all data sets, achieving an area underneath the receiver operating characteristic curve (AUC) for three classes of phenotypes of 0.81 (95%‐confidence interval (CI): 0.65‐0.94) using leave‐one‐out cross‐validation. Besides improving the AUC, our integrative multilevel learning approach led to tighter CIs than using smaller complete predictor data sets (AUC = 0.82 [0.66‐0.94] for boosting). The most important variables for classifying childhood asthma phenotypes comprised novel identified genes, namely PKN2 (protein kinase N2), PTK2 (protein tyrosine kinase 2), and ALPP (alkaline phosphatase, placental).ConclusionOur combination of several data modalities using a novel strategy improved classification of childhood asthma phenotypes but requires validation in external populations. The generic approach is applicable to other multilevel data‐based risk prediction settings, which typically suffer from incomplete data.

Missing Data - Better "Not to Have Them", but What If You Do? (Part 1)

Missing values are ubiquitous in empirical marketing research. If missing data are not dealt with properly, this can lead to a loss of statistical power and distorted parameter estimates. While traditional approaches for handling missing data (e.g., listwise deletion) are still widely used, researchers can nowadays choose among various advanced techniques such as multiple imputation analysis or full-information maximum likelihood estimation. Due to the available software, using these modern missing data methods does not pose a major obstacle. Still, their application requires a sound understanding of the prerequisites and limitations of these methods as well as a deeper understanding of the processes that have led to missing values in an empirical study. This article is Part 1 and first introduces Rubin’s classical definition of missing data mechanisms and an alternative, variable-based taxonomy, which provides a graphical representation. Secondly, a selection of visualization tools available in different R packages for the description and exploration of missing data structures is presented.

A comparison of different methods to handle missing data in the context of propensity score analysis

Propensity score analysis is a popular method to control for confounding in observational studies. A challenge in propensity methods is missing values in confounders. Several strategies for handling missing values exist, but guidance in choosing the best method is needed. In this simulation study, we compared four strategies of handling missing covariate values in propensity matching and propensity weighting. These methods include: complete case analysis, missing indicator method, multiple imputation and combining multiple imputation and missing indicator method. Concurrently, we aimed to provide guidance in choosing the optimal strategy. Simulated scenarios varied regarding missing mechanism, presence of effect modification or unmeasured confounding. Additionally, we demonstrated how missingness graphs help clarifying the missing structure. When no effect modification existed, complete case analysis yielded valid causal treatment effects even when data were missing not at random. In some situations, complete case analysis was also able to partially correct for unmeasured confounding. Multiple imputation worked well if the data were missing (completely) at random, and if the imputation model was correctly specified. In the presence of effect modification, more complex imputation models than default options of commonly used statistical software were required. Multiple imputation may fail when data are missing not at random. Here, combining multiple imputation and the missing indicator method reduced the bias as the missing indicator variable can be a proxy for unobserved confounding. The optimal way to handle missing values in covariates of propensity score models depends on the missing data structure and the presence of effect modification. When effect modification is present, default settings of imputation methods may yield biased results even if data are missing at random.

The HCUP SID Imputation Project: Improving Statistical Inferences for Health Disparities Research by Imputing Missing Race Data.

To identify the most appropriate imputation method for missing data in the HCUP State Inpatient Databases (SID) and assess the impact of different missing data methods on racial disparities research. HCUP SID. A novel simulation study compared four imputation methods (random draw, hot deck, joint multiple imputation [MI], conditional MI) for missing values for multiple variables, including race, gender, admission source, median household income, and total charges. The simulation was built on real data from the SID to retain their hierarchical data structures and missing data patterns. Additional predictive information from the U.S. Census and American Hospital Association (AHA) database was incorporated into the imputation. Conditional MI prediction was equivalent or superior to the best performing alternatives for all missing data structures and substantially outperformed each of the alternatives in various scenarios. Conditional MI substantially improved statistical inferences for racial health disparities research with the SID.

METHODS FOR CLUSTERING TIME SERIES DATA ACQUIRED FROM MOBILE HEALTH APPS.

In our recent Asthma Mobile Health Study (AMHS), thousands of asthma patients across the country contributed medical data through the iPhone Asthma Health App on a daily basis for an extended period of time. The collected data included daily self-reported asthma symptoms, symptom triggers, and real time geographic location information. The AMHS is just one of many studies occurring in the context of now many thousands of mobile health apps aimed at improving wellness and better managing chronic disease conditions, leveraging the passive and active collection of data from mobile, handheld smart devices. The ability to identify patient groups or patterns of symptoms that might predict adverse outcomes such as asthma exacerbations or hospitalizations from these types of large, prospectively collected data sets, would be of significant general interest. However, conventional clustering methods cannot be applied to these types of longitudinally collected data, especially survey data actively collected from app users, given heterogeneous patterns of missing values due to: 1) varying survey response rates among different users, 2) varying survey response rates over time of each user, and 3) non-overlapping periods of enrollment among different users. To handle such complicated missing data structure, we proposed a probability imputation model to infer missing data. We also employed a consensus clustering strategy in tandem with the multiple imputation procedure. Through simulation studies under a range of scenarios reflecting real data conditions, we identified favorable performance of the proposed method over other strategies that impute the missing value through low-rank matrix completion. When applying the proposed new method to study asthma triggers and symptoms collected as part of the AMHS, we identified several patient groups with distinct phenotype patterns. Further validation of the methods described in this paper might be used to identify clinically important patterns in large data sets with complicated missing data structure, improving the ability to use such data sets to identify at-risk populations for potential intervention.

Visually Exploring Missing Values in Multivariable Data Using a Graphical User Interface

Missing values are common in data, and usually require attention in order to conduct the statistical analysis. One of the first steps is to explore the structure of the missing values, and how missingness relates to the other collected variables. This article describes an R package, that provides a graphical user interface (GUI) designed to help explore the missing data structure and to examine the results of different imputation methods. The GUI provides numerical and graphical summaries conditional on missingness, and includes imputations using fixed values, multiple imputations and nearest neighbors.

Illuminate the unknown: evaluation of imputation procedures based on the SAVE survey

Questions about monetary variables (such as income, wealth or savings) are key components of questionnaires on household finances. However, missing information on such sensitive topics is a well-known phenomenon which can seriously bias any inference based only on complete-case analysis. Many imputation techniques have been developed and implemented in several surveys. Using the German SAVE data, a new estimation technique is necessary to overcome the upward bias of monetary variables caused by the initially implemented imputation procedure. The upward bias is the result of adding random draws to the implausible negative values predicted by OLS regressions until all values are positive. To overcome this problem the logarithm of the dependent variable is taken and the predicted values are retransformed to the original scale by Duan’s smearing estimate. This paper evaluates the two different techniques for the imputation of monetary variables implementing a simulation study, where a random pattern of missingness is imposed on the observed values of the variables of interest. A Monte-Carlo simulation based on the observed data shows the superiority of the newly implemented smearing estimate to construct the missing data structure. All waves are consistently imputed using the new method.

Predicting missing biomarker data in a longitudinal study of Alzheimer disease

To investigate predictors of missing data in a longitudinal study of Alzheimer disease (AD). The Alzheimer's Disease Neuroimaging Initiative (ADNI) is a clinic-based, multicenter, longitudinal study with blood, CSF, PET, and MRI scans repeatedly measured in 229 participants with normal cognition (NC), 397 with mild cognitive impairment (MCI), and 193 with mild AD during 2005-2007. We used univariate and multivariable logistic regression models to examine the associations between baseline demographic/clinical features and loss of biomarker follow-ups in ADNI. CSF studies tended to recruit and retain patients with MCI with more AD-like features, including lower levels of baseline CSF Aβ(42). Depression was the major predictor for MCI dropouts, while family history of AD kept more patients with AD enrolled in PET and MRI studies. Poor cognitive performance was associated with loss of follow-up in most biomarker studies, even among NC participants. The presence of vascular risk factors seemed more critical than cognitive function for predicting dropouts in AD. The missing data are not missing completely at random in ADNI and likely conditional on certain features in addition to cognitive function. Missing data predictors vary across biomarkers and even MCI and AD groups do not share the same missing data pattern. Understanding the missing data structure may help in the design of future longitudinal studies and clinical trials in AD.

Illuminate the Unknown: Evaluation of Imputation Procedures Based on the Save Survey

Questions about monetary variables (such as income, wealth or savings) are key components of questionnaires on household finances. However, missing information on such sensitive topics is a well-known phenomenon which can seriously bias any inference based only on complete cases analysis. Many imputation techniques have been developed and implemented in several surveys. Using the German SAVE data, this paper evaluates different techniques for the imputation of monetary variables implementing a simulation study, where a random pattern of missingness is imposed on the observed values of the variables of interest. New estimation techniques are necessary to overcome the upward bias of monetary variables caused by the initially implemented imputation procedure. A Monte-Carlo simulation based on the observed data shows the superiority of the newly implemented smearing estimate to construct the missing data structure. All waves are consistently imputed using the new method.