Missing Data Patterns Research Articles

Impact of missing data and ICC on full information maximum-likelihood estimation in multilevel SEMs

A Monte Carlo simulation study was conducted to investigate the performance of full information maximum-likelihood (FIML) estimator in multilevel structural equation modeling (SEM) with missing data and different intra-class correlations (ICCs) coefficients. The study simulated the influence of two independent variables (missing data patterns, and ICC coefficients) in multilevel SEM on five outcome measures (model rejection rates, parameter estimate bias, standard error bias, coverage, and power). Results indicated that FIML parameter estimates were generally robust for data missing on outcomes and/or higher-level predictor variables under the data completely at random (MCAR) and for data missing at random (MAR). However, FIML estimation yielded substantially lower parameter and standard error bias when data was not missing on higher-level variables, and in high rather than in low ICC conditions (0.50 vs 0.20). Future research should extend to further examination of the impacts of data distribution, complexity of the between-level model, and missingness on the between-level variables on FIML estimation performance.

A novel machine learning-based imputation strategy for missing data in step-stress accelerated degradation test

The presence of missing data is a significant data quality issue that negatively impacts the accuracy and reliability of data analysis. This issue is especially relevant in the context of accelerated tests, particularly for step-stress accelerated degradation tests. While missing data can occur due to objective factors or human error, high missing rate is an inevitable pattern of missing data that will occur during the conversion process of accelerated test data. This type of missing data manifests as a degradation dataset with unequal measuring intervals. Therefore, developing a more appropriate imputation method for accelerated test data is essential. In this study, we propose a novel hybrid imputation method that combines the LSSVM and RBF models to address missing data problems. A comparison is conducted between the proposed model and various traditional and machine learning imputation methods using simulation data, to justify the advantages of the proposed model over the existing methods. Finally, the proposed model is implemented on real degradation datasets of the super-luminescent diode (SLD) to validate its performance and effectiveness in dealing with missing data in step-stress accelerated degradation test. Additionally, due to the generalizability of the proposed method, it is expected to be applicable in other scenarios with high missing data rates.

Link prediction accuracy on real-world networks under non-uniform missing-edge patterns.

Real-world network datasets are typically obtained in ways that fail to capture all edges. The patterns of missing data are often non-uniform as they reflect biases and other shortcomings of different data collection methods. Nevertheless, uniform missing data is a common assumption made when no additional information is available about the underlying missing-edge pattern, and link prediction methods are frequently tested against uniformly missing edges. To investigate the impact of different missing-edge patterns on link prediction accuracy, we employ 9 link prediction algorithms from 4 different families to analyze 20 different missing-edge patterns that we categorize into 5 groups. Our comparative simulation study, spanning 250 real-world network datasets from 6 different domains, provides a detailed picture of the significant variations in the performance of different link prediction algorithms in these different settings. With this study, we aim to provide a guide for future researchers to help them select a link prediction algorithm that is well suited to their sampled network data, considering the data collection process and application domain.

Addressing data gaps in environmental sensors: a case study on temporal patterns and restoration strategies in East Kazakhstan region

Nowadays, the monitoring systems embedded with diverse sensors within the Internet of Things play an important role in how we collect data and provide in-depth insights into human behavior, environmental trends, and different industrial processes. These sensors can gather data in real-time and due to their ability to connect through various communication interfaces, it is possible to integrate such devices into existing networks ensuring that data can be transmitted in real-time to central systems or mobile devices. However, many factors such as connectivity issues, severe weather, and sensor faults, might cause missing data, thus leading to poor decision-making. In this study, an in-depth analysis was performed on environmental meteorological data collected from May 2022 to March 2023 in the East Kazakhstan region using Vaisala WXT536 multi-parameter weather sensor. The data restoration strategy that is based on the duration of data gaps is evaluated. For short-term gaps under 30 minutes in duration, the mean and median imputation achieved higher results with a root mean squared error of 0.25 for temperature than training Random Forest algorithm which achieved 3.59, while for the extended periods of missing values, the Long Short-Term Memory model was utilized with 2.47 for the temperature. The main goal of the research is to analyze the temporal patterns of missing data and potential reasons for these failures as well as to evaluate the data restoration strategies.

A data analysis framework for combining multiple batches increases the power of isobaric proteomics experiments.

We present a framework for the analysis of multiplexed mass spectrometry proteomics data that reduces estimation error when combining multiple isobaric batches. Variations in the number and quality of observations have long complicated the analysis of isobaric proteomics data. Here we show that the power to detect statistical associations is substantially improved by utilizing models that directly account for known sources of variation in the number and quality of observations that occur across batches.In a multibatch benchmarking experiment, our open-source software (msTrawler) increases the power to detect changes, especially in the range of less than twofold changes, while simultaneously increasing quantitative proteome coverage by utilizing more low-signal observations. Further analyses of previously published multiplexed datasets of 4 and 23 batches highlight both increased power and the ability to navigate complex missing data patterns without relying on unverifiable imputations or discarding reliable measurements.

Sample composition and representativeness on Understanding Society

AbstractIn this paper, we provide an overview of the sample design of Understanding Society and the consequent nature of design weights as well as a description of procedures that are implemented in order to maximise participation by sample members and procedures that are implemented to produce non‐response adjustments to the design weights. We then present some indicators of sample representativeness at the initial wave and of the impact that subsequent sample attrition has on this before concluding with some reflections on the nature of representativeness and estimation methods in the context of a highly complex sample design and complex patterns of missing data arising from non‐response.

Towards real-world traffic prediction and data imputation: A multi-task pretraining and fine-tuning approach

Real-world traffic prediction is challenging and requires accuracy, efficiency, and generalizability for applications. Most studies used two-step data imputation-prediction procedures or recurrent networks to perform predictions based on imputed data, resulting in error accumulation. Training the same model independently for different missing rates and missing data patterns requires high computational costs. This strategy cannot be integrated with existing methods, causing low generalizability. We propose a multi-task pretraining and fine-tuning (MTPF) approach to address these issues. Specifically, we pretrain an encoder and decoders with different missing data patterns and missing rates and an adversarial noise to learn the correlations between different scenarios and generate robust hidden representations. The encoder and decoder perform fine-tuning for specific missing data scenarios after pretraining. Multiple parallel tasks, including prediction, data imputation, contrastive learning, and adaptive graph learning, are used for pretraining to enable the model to learn the hidden representation for prediction and data imputation during fine-tuning. The MTPF is compared with nine baseline models for prediction and five baseline models for data imputation using two real-world datasets. The experimental results show that MTPF outperforms many state-of-the-art baseline models, has a shorter training time, and is a model-agnostic off-the-shelf plug-in that improves baseline model performance.

Variable selection for high-dimensional incomplete data

Regression analysis is often affected by high dimensionality, severe multicollinearity, and a large proportion of missing data. These problems may mask important relationships and even lead to biased conclusions. This paper proposes a novel computationally efficient method that integrates data imputation and variable selection to address these issues. More specifically, the proposed method incorporates a new multiple imputation algorithm based on matrix completion (Multiple Accelerated Inexact Soft-Impute), a more stable and accurate new randomized lasso method (Hybrid Random Lasso), and a consistent method to integrate a variable selection method with multiple imputation. Compared to existing methodologies, the proposed approach offers greater accuracy and consistency through mechanisms that enhances robustness against different missing data patterns and sampling variations. The method is applied to analyze the Asian American minority subgroup in the 2017 National Youth Risk Behavior Survey, where key risk factors related to the intention for suicide among Asian Americans are studied. Through simulations and real data analyses on various regression and classification settings, the proposed method demonstrates enhanced accuracy, consistency, and efficiency in both variable selection and prediction.

Never miss a beep: Using mobile sensing to investigate (non-)compliance in experience sampling studies

Given the increasing number of studies in various disciplines using experience sampling methods, it is important to examine compliance biases because related patterns of missing data could affect the validity of research findings. In the present study, a sample of 592 participants and more than 25,000 observations were used to examine whether participants responded to each specific questionnaire within an experience sampling framework. More than 400 variables from the three categories of person, behavior, and context, collected multi-methodologically via traditional surveys, experience sampling, and mobile sensing, served as predictors. When comparing different linear (logistic and elastic net regression) and non-linear (random forest) machine learning models, we found indication for compliance bias: response behavior was successfully predicted. Follow-up analyses revealed that study-related past behavior, such as previous average experience sampling questionnaire response rate, was most informative for predicting compliance, followed by physical context variables, such as being at home or at work. Based on our findings, we discuss implications for the design of experience sampling studies in applied research and future directions in methodological research addressing experience sampling methodology and missing data.

A modified Nadaraya–Watson procedure for variable selection and nonparametric prediction with missing data

We propose a new method for variable selection and prediction under a nonparametric regression setting, where a covariate may be missing either because its value is hidden from the observer or because it is inapplicable to the particular subject being observed. Despite its practical relevance, the problem has received little attention in the literature and its solutions are largely non-existent. Our proposal hinges on the construction of a modified Nadaraya–Watson estimator of the conditional mean regression function, with its bandwidths regularised to select variables and its weights adapted to accommodate different types of missingness. The method allows for information sharing across different missing data patterns without affecting consistency of the estimator. Unlike other conventional methods such as those based on imputations or likelihoods, our method requires only mild assumptions on the model and the missingness mechanism. For prediction we focus on finding relevant variables for predicting mean responses, conditional on covariate vectors subject to a given type of missingness. Our theoretical and numerical results show that the new method is consistent in variable selection and yields better prediction accuracy compared to existing methods.

GCN-ST-MDIR: Graph Convolutional Network-Based Spatial-Temporal Missing Air Pollution Data Pattern Identification and Recovery

Missing data pattern identification and recovery (MDIR) is vital for accurate air pollution monitoring. To recover the missing air pollution data, GCN-ST-MDIR, a Graph Convolutional Network (GCN)-based MDIR framework, is proposed to identify daily missing data patterns and automatically select the best recovery method. GCN-ST-MDIR presents four novelties: (1) A new graph construction is developed to improve GCN data representation for MDIR using S-T similarity matrix and domain-specific knowledge (e.g. weekend/weekday). (2) A TL component is used to pre-train LSCE and ILSCE models. (3) A GCN structure outputs a selection indicator to determine the dominant missing pattern for daily input. The pre-trained data recovery model's accuracy is incorporated into the GCN loss function to penalize the wrong indicator. (4) The output of the GCN structure is used as a score to combine LSCE and ILSCE. Results show that the domain-specific S-T regularity and irregularity can be used as the prior information for both GCN and ILSCE/LSCE to enhance feature extraction. Our model considerably improves the recovery performance as compared to the baselines. GCN-ST-MDIR has achieved an accuracy of 88.48% for general missing data recovery with consecutively and sporadically missing data. GCN-ST-MDIR can be extended to many other S-T MDIR challenges.

Handling missing values in the analysis of between-hospital differences in ordinal and dichotomous outcomes: a simulation study

Missing data are frequently encountered in registries that are used to compare performance across hospitals. The most appropriate method for handling missing data when analysing differences in outcomes between hospitals...

Addressing missing data in the estimation of time-varying treatments in comparative effectiveness research.

Comparative effectiveness research is often concerned with evaluating treatment strategies sustained over time, that is, time-varying treatments. Inverse probability weighting (IPW) is often used to address the time-varying confounding by re-weighting the sample according to the probability of treatment receipt at each time point. IPW can also be used to address any missing data by re-weighting individuals according to the probability of observing the data. The combination of these two distinct sets of weights may lead to inefficient estimates of treatment effects due to potentially highly variable total weights. Alternatively, multiple imputation (MI) can be used to address the missing data by replacing each missing observation with a set of plausible values drawn from the posterior predictive distribution of the missing data given the observed data. Recent studies have compared IPW and MI for addressing the missing data in the evaluation of time-varying treatments, but they focused on missing confounders and monotone missing data patterns. This article assesses the relative advantages of MI and IPW to address missing data in both outcomes and confounders measured over time, and across monotone and non-monotone missing data settings. Through a comprehensive simulation study, we find that MI consistently provided low bias and more precise estimates compared to IPW across a wide range of scenarios. We illustrate the implications of method choice in an evaluation of biologic drugs for patients with severe rheumatoid arthritis, using the US National Databank for Rheumatic Diseases, in which 25% of participants had missing health outcomes or time-varying confounders.

High-Dimensional Imputation for the Social Sciences: A Comparison of State-of-The-Art Methods

Including a large number of predictors in the imputation model underlying a multiple imputation (MI) procedure is one of the most challenging tasks imputers face. A variety of high-dimensional MI techniques can help, but there has been limited research on their relative performance. In this study, we investigated a wide range of extant high-dimensional MI techniques that can handle a large number of predictors in the imputation models and general missing data patterns. We assessed the relative performance of seven high-dimensional MI methods with a Monte Carlo simulation study and a resampling study based on real survey data. The performance of the methods was defined by the degree to which they facilitate unbiased and confidence-valid estimates of the parameters of complete data analysis models. We found that using lasso penalty or forward selection to select the predictors used in the MI model and using principal component analysis to reduce the dimensionality of auxiliary data produce the best results.

A standardized workflow for long-term longitudinal actigraphy data processing using one year of continuous actigraphy from the CAN-BIND Wellness Monitoring Study

Monitoring sleep and activity through wearable devices such as wrist-worn actigraphs has the potential for long-term measurement in the individual’s own environment. Long periods of data collection require a complex approach, including standardized pre-processing and data trimming, and robust algorithms to address non-wear and missing data. In this study, we used a data-driven approach to quality control, pre-processing and analysis of longitudinal actigraphy data collected over the course of 1 year in a sample of 95 participants. We implemented a data processing pipeline using open-source packages for longitudinal data thereby providing a framework for treating missing data patterns, non-wear scoring, sleep/wake scoring, and conducted a sensitivity analysis to demonstrate the impact of non-wear and missing data on the relationship between sleep variables and depressive symptoms. Compliance with actigraph wear decreased over time, with missing data proportion increasing from a mean of 4.8% in the first week to 23.6% at the end of the 12 months of data collection. Sensitivity analyses demonstrated the importance of defining a pre-processing threshold, as it substantially impacts the predictive value of variables on sleep-related outcomes. We developed a novel non-wear algorithm which outperformed several other algorithms and a capacitive wear sensor in quality control. These findings provide essential insight informing study design in digital health research.

Road network pixelization: A traffic flow imputation method based on image restoration techniques

The complete traffic network data is crucial for accurate traffic data prediction in intelligent transportation systems. Inspired by the success of Generative Adversarial Networks (GANs) in image restoration, this study proposes a traffic flow imputation method by employing image restoration techniques. First, we propose a trajectory data representation method called Trajectory2Matrix, that converts the trajectory data into a two-dimensional spatiotemporal relation feature map. Consequently, the data imputation scale and scope are increased. Second, a spatiotemporal feature map generation module based on a graph convolutional network is designed to optimize the GAN generator, thus utilizing its advantages for non-Euclidean data and dynamic spatiotemporal correlation. Finally, a heterogeneous multisource data fusion module based on a channel attention mechanism is proposed to merge dynamic/static external attributes and multimode characteristics in time. The proposed method improved imputation accuracy and robustness for multitype missing data patterns, especially in high missing rate situations.

Engineering Traffic Prediction With Online Data Imputation: A Graph-Theoretic Perspective

Accurate and timely prediction about the current and near-term future traffic conditions is one of the effective ways to relieve traffic congestion and improve the operation efficiency of road networks. However, the missing data problem is inevitable when collecting real-time traffic flow information due to lossy communication and detector malfunction, which significantly affects the accuracy of prediction. To address this problem, in this article, we propose an approach to exploit online traffic prediction considering missing data effects. Unlike most existing methods imputing all the missing data before traffic prediction, the proposed strategy takes the spatio-temporal relationships between missing data and observed data into account to optimize data imputation patterns, thereby improving the efficiency of online traffic prediction. Specifically, we first propose an analytical framework for online traffic prediction with missing data by extending the space-time autoregressive integrated moving average model to incorporate missing data effects. Then, based on the combined use of optimal cut and data imputation optimization, a graph-theoretic technique is presented to determine the imputed data with consideration of road network topology and missing data pattern. Finally, experiments are conducted based on two real-world datasets. Experimental results indicate the superiority of the proposed approach in accuracy and efficiency compared with existing strategies of traffic flow prediction with missing data, particularly under the circumstance of high data missing ratios in urban road networks.

Phylogeny of Species, Infraspecific Taxa, and Forms in Iris Subgenus Xiphium (Iridaceae), From the Mediterranean Basin Biodiversity Hotspot

Abstract— Iris subgenus Xiphium is a small group of taxa that occur in the Mediterranean Basin, a long-recognized biodiversity hotspot. Phylogenetic relationships among these Iris were reconstructed based on sequence data from 110 nuclear markers (coding regions) and plastomes using Bayesian inference and maximum likelihood methods. Best trees based on plastome and combined datasets resolved Iris subgenus Xiphium and I. xiphium as not monophyletic while nuclear and all-gene (combined nuclear and plastid coding regions) datasets resolved the subgenus as monophyletic but I. xiphium as not monophyletic. Topology tests indicated the alternative hypothesis of a monophyletic subgenus is not rejected by the combined dataset while a monophyletic I. xiphium is rejected. Topology tests also showed the hypothesis of a polyphyletic subgenus is rejected by the nuclear and all-gene datasets, however, a monophyletic I. xiphium is not rejected by these two datasets. We hypothesize that the subgenus is monophyletic based on these analyses, morphology, and biogeography, and that uneven patterns of missing data is a likely reason for topological incongruence among datasets. A previously suggested informal group within the subgenus was supported. Patterns of relationships among species suggest multiple exchanges between the African and European continents but also the importance of the Strait of Gibraltar as a barrier to genetic exchange.

High-dimensional conditionally Gaussian state space models with missing data

We develop an efficient sampling approach for handling complex missing data patterns and a large number of missing observations in conditionally Gaussian state space models. Two important examples are dynamic factor models with unbalanced datasets and large Bayesian VARs with variables in multiple frequencies. A key observation underlying the proposed approach is that the joint distribution of the missing data conditional on the observed data is Gaussian. Furthermore, the inverse covariance or precision matrix of this conditional distribution is sparse, and this special structure can be exploited to substantially speed up computations. We illustrate the methodology using two empirical applications. The first application combines quarterly, monthly and weekly data using a large Bayesian VAR to produce weekly GDP estimates. In the second application, we extract latent factors from unbalanced datasets involving over a hundred monthly variables via a dynamic factor model with stochastic volatility.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

ObjectiveVital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development. Study Design and SettingElectronic medical record (EMR) data from five Australian hospitals (1 January 2019–31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots. ResultsThe data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor. ConclusionSummary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.