Non-random Missing Data Research Articles

Combining electronic health records data from a clinical research network with registry data to examine long-term outcomes of interventions and devices: an observational cohort study

ObjectivesTo assess the feasibility of assessing long-term outcomes of peripheral vascular intervention (PVI) by linking data from a clinical registry to electronic health records (EHR) data from a clinical research...

2198 Evidence of systematic missingness in frailty data: a European cohort study

Abstract Introduction Emergency department (ED) frailty screening is recommended in guidelines for its potential to trigger earlier and more appropriate comprehensive evaluation and intervention for the most vulnerable patients. Post-implementation studies of the Clinical Frailty Scale (CFS) typically observe around 50% concordance with screening. Little is known regarding the characteristics of those people omitted. Methods The Frailty in European Emergency Departments (FEED) cohort study observed prevalence of frailty, administering the CFS to consecutive attenders over twenty-four hours. Retrospective ‘normal day’ data from two weeks prior were also collected, where sites used retrievable electronic health records. Age, sex, ethnic group, mode and time of arrival and departure, NEWS2 score, and use of resuscitation areas were recorded. CFS missingness was assessed for distribution and dependency with other variables using chi-squared tests. The frailty distributions in prospective and retrospective data were compared with the Kruskal-Wallis test. Results Only five of sixty-two sites collected CFS scores in retrievable electronic records. The cohorts included 368 individuals prospectively and 399 retrospectively. At these sites, 14% prospective and 55% retrospective CFS observations were missing. CFS entries were more frequently missing in people with non-white ethnic group (p = 0.007) and self-presentation (p < 0.001). The distributions of CFS differed significantly (p = 0.009); on the retrospective day, no individuals were assigned CFS scores 1 or 9, and CFS scores 4 and 6 were over-represented. Conclusion Acknowledging the limited participation and use of snapshot data, these findings alert the presence of systematic, non-random missing data in routine CFS screening. Systematic missingness in frailty data has critical implications for research in geriatric emergency medicine, presenting real limitations in validity where studies seek to analyse routinely collected data to reach representative inferences. Screening practices and retrievability of data warrant further study and improvement.

Cross-modal missing time-series imputation using dense spatio-temporal transformer nets.

Due to irregular sampling or device failure, the data collected from sensor network has missing value, that is, missing time-series data occurs. To address this issue, many methods have been proposed to impute random or non-random missing data. However, the imputation accuracy of these methods are not accurate enough to be applied, especially in the case of complete data missing (CDM). Thus, we propose a cross-modal method to impute time-series missing data by dense spatio-temporal transformer nets (DSTTN). This model embeds spatial modal data into time-series data by stacked spatio-temporal transformer blocks and deployment of dense connections. It adopts cross-modal constraints, a graph Laplacian regularization term, to optimize model parameters. When the model is trained, it recovers missing data finally by an end-to-end imputation pipeline. Various baseline models are compared by sufficient experiments. Based on the experimental results, it is verified that DSTTN achieves state-of-the-art imputation performance in the cases of random and non-random missing. Especially, the proposed method provides a new solution to the CDM problem.

Sampling Importance Resampling Algorithm with Nonignorable Missing Response Variable Based on Smoothed Quantile Regression

The presence of nonignorable missing response variables often leads to complex conditional distribution patterns that cannot be effectively captured through mean regression. In contrast, quantile regression offers valuable insights into the conditional distribution. Consequently, this article places emphasis on the quantile regression approach to address nonrandom missing data. Taking inspiration from fractional imputation, this paper proposes a novel smoothed quantile regression estimation equation based on a sampling importance resampling (SIR) algorithm instead of nonparametric kernel regression methods. Additionally, we present an augmented inverse probability weighting (AIPW) smoothed quantile regression estimation equation to reduce the influence of potential misspecification in a working model. The consistency and asymptotic normality of the empirical likelihood estimators corresponding to the above estimating equations are proven under the assumption of a correctly specified parameter working model. Furthermore, we demonstrate that the AIPW estimation equation converges to an IPW estimation equation when a parameter working model is misspecified, thus illustrating the robustness of the AIPW estimation approach. Through numerical simulations, we examine the finite sample properties of the proposed method when the working models are both correctly specified and misspecified. Furthermore, we apply the proposed method to analyze HIV—CD4 data, thereby exploring variations in treatment effects and the influence of other covariates across different quantiles.

Measuring party system institutionalization in democracies

Party system institutionalization (PSI) is regarded as a critical underpinning of democracy. However, the systematic study of PSI in democracies is constrained by weaknesses in existing measures, which are limited in coverage or comprehensiveness, and do not account for the latent nature of the concept, measurement error, and non-random missing data. This article presents a novel measure of PSI that uses a Bayesian latent variable measurement strategy to overcome extant measurement issues. The subsequent measure not only offers unmatched coverage and has demonstrated validity, but also exhibits more robust empirical associations with a range of outcomes related to the performance of democracy than existing measures. The measure should facilitate more integrated research on the causes and consequences of PSI in democracies around the world.

Bridging the Data Gap: Enhancing the Spatiotemporal Accuracy of Hourly PM2.5 Concentration through the Fusion of Satellite-Derived Estimations and Station Observations

Satellite-derived aerosol optical depth (AOD) has been extensively utilized for retrieving ground-level PM2.5 distributions. However, the presence of non-random missing data gaps in AOD poses a challenge to directly obtaining the gap-free AOD-derived PM2.5, thereby impeding accurate exposure risk assessment. Here, this study presents a novel and flexible framework that couples stacking and flexible spatiotemporal data fusion (FSDAF) approaches. By integrating multiple models and data sources, this framework aims to generate hourly (24-h) gap-free PM2.5 estimates for the Beijing–Tianjin–Hebei (BTH) region in 2018. This study effectively reconstructed data at least three times more effectively than the original AOD-derived PM2.5, achieving the Pearson coefficient (r), the coefficient determination (R2), root mean squared error (RMSE), and mean absolute error (MAE) values of 0.91, 0.84, 19.38 µg/m3, and 12.17 µg/m3, respectively, based on entire samples. Such strong predictive performance was also exhibited in spatial-based (r: 0.92–0.93, R2: 0.85–0.87, RMSE: 18.13 µg/m3–20.18 µg/m3, and MAE: 11.21 µg/m3–12.52 µg/m3) and temporal-based (r: 0.91–0.98, R2: 0.82–0.96, RMSE: 3.8 µg/m3–21.89 µg/m3, and MAE: 2.71 µg/m3–14.00 µg/m3) validations, indicating the robustness of this framework. Additionally, this framework enables the assessment of annual and seasonal PM2.5 concentrations and distributions, revealing that higher levels are experienced in the southern region, while lower levels prevail in the northern part. Winter exhibits the most severe levels, followed by spring and autumn, with comparatively lower levels in summer. Notably, the proposed framework effectively mitigates bias in calculating population-weighted exposure risk by filling data gaps with calculated values of 51.04 µg/m3, 54.17 µg/m3, 56.24 µg/m3, and 55.00 µg/m3 in Beijing, Tianjin, Hebei, and the BTH region, respectively.

Evaluating Policies Early in a Pandemic: Bounding Policy Effects with Nonrandomly Missing Data

Abstract During the early part of the Covid-19 pandemic, national and local governments introduced a number of policies to combat the spread of Covid-19. In this paper, we propose a new approach to bound the effects of such early-pandemic policies on Covid-19 cases and other outcomes while dealing with complications arising from (i) limited availability of Covid-19 tests, (ii) differential availability of Covid-19 tests across locations, and (iii) eligibility requirements for individuals to be tested. We use our approach study the effects of Tennessee's expansion of Covid-19 testing early in the pandemic and find that the policy decreased Covid-19 cases.

The role of mothers on female labour force participation: an approach using historical parish records

Using Portuguese parish data from 1675 to 1925, we estimate the relationship between a mother’s participation in the labour force and that of her daughter. We adapt a methodology to prevent bias that originates from potentially non-random missing data. Ignoring the missingness process results in substantial downward-biased estimates of the relationship, even for a proportion of missing values as low as 20 per cent. In contrast, our methodology yields unbiased estimates regardless of the proportion of missing values. We document the existence of a strong, positive association between the mother’s participation and that of her daughter long before the twentieth century’s substantial changes in education and the labour market

A Hybrid Imputation Method for Multi-Pattern Missing Data: A Case Study on Type II Diabetes Diagnosis

Real medical datasets usually consist of missing data with different patterns which decrease the performance of classifiers used in intelligent healthcare and disease diagnosis systems. Many methods have been proposed to impute missing data, however, they do not fulfill the need for data quality especially in real datasets with different missing data patterns. In this paper, a four-layer model is introduced, and then a hybrid imputation (HIMP) method using this model is proposed to impute multi-pattern missing data including non-random, random, and completely random patterns. In HIMP, first, non-random missing data patterns are imputed, and then the obtained dataset is decomposed into two datasets containing random and completely random missing data patterns. Then, concerning the missing data patterns in each dataset, different single or multiple imputation methods are used. Finally, the best-imputed datasets gained from random and completely random patterns are merged to form the final dataset. The experimental evaluation was conducted by a real dataset named IRDia including all three missing data patterns. The proposed method and comparative methods were compared using different classifiers in terms of accuracy, precision, recall, and F1-score. The classifiers’ performances show that the HIMP can impute multi-pattern missing values more effectively than other comparative methods.

The Association Between Epigenetic Clocks and Physical Functioning in Older Women: A 3-Year Follow-up.

BackgroundEpigenetic clocks are composite markers developed to predict chronological age or mortality risk from DNA methylation (DNAm) data. The present study investigated the associations between 4 epigenetic clocks (Horvath’s and Hannum’s DNAmAge and DNAm GrimAge and PhenoAge) and physical functioning during a 3-year follow-up.MethodWe studied 63- to 76-year-old women (N = 413) from the Finnish Twin Study on Aging. DNAm was measured from blood samples at baseline. Age acceleration (AgeAccel), that is, discrepancy between chronological age and DNAm age, was determined as residuals from linear model. Physical functioning was assessed under standardized laboratory conditions at baseline and at follow-up. A cross-sectional analysis was performed with path models, and a longitudinal analysis was conducted with repeated measures linear models. A nonrandom missing data analysis was performed.ResultsIn comparison to the other clocks, GrimAgeAccel was more strongly associated with physical functioning. At baseline, GrimAgeAccel was associated with lower performance in the Timed Up and Go (TUG) test and the 6-minute walk test. At follow-up, significant associations were observed between GrimAgeAccel and lowered performance in the TUG, 6-minute and 10-m walk tests, and knee extension and ankle plantar flexion strength tests.ConclusionsThe DNAm GrimAge, a novel estimate of biological aging, associated with decline in physical functioning over the 3-year follow-up in older women. However, associations between chronological age and physical function phenotypes followed similar pattern. Current epigenetic clocks do not provide strong benefits in predicting the decline of physical functioning at least during a rather short follow-up period and restricted age range.

Nonrandom missing data can bias Principal Component Analysis inference of population genetic structure.

Population genetic studies in non-model systems increasingly use next-generation sequencing to obtain more loci, but such methods also generate more missing data that may affect downstream analyses. Here we focus on the principal component analysis (PCA) which has been widely used to explore and visualize population structure with mean-imputed missing data. We simulated data of different population models with various total missingness (1%, 10%, 20%) introduced either randomly or biased among individuals or populations. We found that individuals biased with missing data would be dragged away from their real population clusters to the origin of PCA plots, making them indistinguishable from true admixed individuals and potentially leading to misinterpreted population structure. We also generated empirical data of the big brown bat (Eptesicus fuscus) using restriction site-associated DNA sequencing (RADseq). We filtered three data sets with 19.12%, 9.87%, and 1.35% total missingness, all showing nonrandom missing data with biased individuals dragged towards the PCA origin, consistent with results from simulations. We highlight the importance of considering missing data effects on PCA in non-model systems where nonrandom missing data are common due to varying sample quality. To help detect missing data effects, we suggest to (1) plot PCA with a colour gradient showing per sample missingness, (2) interpret samples close to the PCA origin with extra caution, (3) explore filtering parameters with and without the missingness-biased samples, and (4) use complementary analyses (e.g., model-based methods) to cross-validate PCA results and help interpret population structure.

Multiple imputation and direct estimation for qPCR data with non-detects

BackgroundQuantitative real-time PCR (qPCR) is one of the most widely used methods to measure gene expression. An important aspect of qPCR data that has been largely ignored is the presence of non-detects: reactions failing to exceed the quantification threshold and therefore lacking a measurement of expression. While most current software replaces these non-detects with a value representing the limit of detection, this introduces substantial bias in the estimation of both absolute and differential expression. Single imputation procedures, while an improvement on previously used methods, underestimate residual variance, which can lead to anti-conservative inference.ResultsWe propose to treat non-detects as non-random missing data, model the missing data mechanism, and use this model to impute missing values or obtain direct estimates of model parameters. To account for the uncertainty inherent in the imputation, we propose a multiple imputation procedure, which provides a set of plausible values for each non-detect. We assess the proposed methods via simulation studies and demonstrate the applicability of these methods to three experimental data sets. We compare our methods to mean imputation, single imputation, and a penalized EM algorithm incorporating non-random missingness (PEMM). The developed methods are implemented in the R/Bioconductor package nondetects.ConclusionsThe statistical methods introduced here reduce discrepancies in gene expression values derived from qPCR experiments in the presence of non-detects, providing increased confidence in downstream analyses.

Handling Missing Data in the Short Form-12 Health Survey (SF-12): Concordance of Real Patient Data and Data Estimated by Missing Data Imputation Procedures.

If information on single items in the Short Form–12 health survey (SF-12) is missing, the analysis of only complete cases causes a loss of statistical power and, in case of nonrandom missing data (MD), systematic bias. This study aimed at evaluating the concordance of real patient data and data estimated by different MD imputation procedures in the items of the SF-12 assessment. For this ends, MD were examined in a sample of 1,137 orthopedic patients. Additionally, MD were simulated (a) in the subsample of orthopedic patients exhibiting no MD (n = 810; 71%) as well as (b) in a sample of 6,970 respondents representing the German general population (95.8% participants with complete data) using logistic regression modelling. Simulated MD were replaced by mean values as well as regression-, expectation-maximization- (EM-), and multiple imputation estimates. Higher age and lower education were associated with enhanced probabilities of MD. In terms of accuracy in both data sets, the EM-procedure (ICC2,1 = .33-.72) outperformed alternative estimation approaches substantially (e.g., regression imputation: ICC2,1 = .18-.48). The EM-algorithm can be recommended to estimate MD in the items of the SF-12, because it reproduces the actual patient data most accurately.

Identification of abnormal conditions in high-dimensional chemical process based on feature selection and deep learning

Abstract Identification of abnormal conditions is essential in the chemical process. With the rapid development of artificial intelligence technology, deep learning has attracted a lot of attention as a promising fault identification method in chemical process recently. In the high-dimensional data identification using deep neural networks, problems such as insufficient data and missing data, measurement noise, redundant variables, and high coupling of data are often encountered. To tackle these problems, a feature based deep belief networks (DBN) method is proposed in this paper. First, a generative adversarial network (GAN) is used to reconstruct the random and non-random missing data of chemical process. Second, the feature variables are selected by Spearman's rank correlation coefficient (SRCC) from high-dimensional data to eliminate the noise and redundant variables and, as a consequence, compress data dimension of chemical process. Finally, the feature filtered data is deeply abstracted, learned and tuned by DBN for multi-case fault identification. The application in the Tennessee Eastman (TE) process demonstrates the fast convergence and high accuracy of this proposal in identifying abnormal conditions for chemical process, compared with the traditional fault identification algorithms.

Parameter estimation in spatial econometric models with non-random missing data

ABSTRACT This study examines the problem of parameter estimation in spatial econometric/social interaction models with non-random missing outcome data. First, we construct a sample selection model considering spatial lag (autoregressive) dependence. Then, we suggest a parameter estimation method for this model by slightly modifying the Bayesian Markov chain Monte Carlo algorithm proposed in an existing study. A simple illustration indicates that the proposed parameter estimation method performs well overall if the spatial autocorrelation is moderate (spatial parameter equals 0.5 or less), even under a relatively high missing data ratio (around 40%).

ESTIMATION AND INFERENCE IN METABOLOMICS WITH NON-RANDOM MISSING DATA AND LATENT FACTORS.

High throughput metabolomics data are fraught with both non-ignorable missing observations and unobserved factors that influence a metabolite's measured concentration, and it is well known that ignoring either of these complications can compromise estimators. However, current methods to analyze these data can only account for the missing data or unobserved factors, but not both. We therefore developed MetabMiss, a statistically rigorous method to account for both non-random missing data and latent factors in high throughput metabolomics data. Our methodology does not require the practitioner specify a likelihood for the missing data, and makes investigating the relationship between the metabolome and tens, or even hundreds, of phenotypes computationally tractable. We demonstrate the fidelity of Metab-Miss's estimates using both simulated and real metabolomics data, and prove their asymptotic correctness when the sample size and number of metabolites grows to infinity.

Extreme gradient boosting model to estimate PM2.5 concentrations with missing-filled satellite data in China

Several studies have attempted to predict ground PM2.5 concentrations using satellite aerosol optical depth (AOD) retrieval. However, over 70%–90% of aerosol retrievals are non-random missing, which limits and biases the estimation. To the best of our knowledge, this issue has not been well resolved to date.The aim of this study was to develop an interpolation technique to handle the missing data retrieval problem and to estimate the daily PM2.5 for a high coverage dataset with 3-km resolution in China by fitting the complex temporal and spatial variations.We developed a two-step interpolation method (i.e., the mixed-effect model and inverse distance weighting technology) to replace the missing values in AOD. Next, the extreme gradient boosting (XGBoost) technique that includes a non-linear exposure-lag-response model (NELRM) was proposed and validated to estimate the daily levels of PM2.5 across China during 2014–2015.After two steps of interpolation, the missing value rate of daily AOD data was reduced from 87.91% to 13.83%. The cross-validation (CV) R-square, root mean square error (RMSE) and mean absolute percentage prediction error (MAPE) of the interpolation were 0.76, 0.10 and 21.41%, respectively. The cross-validation for the prediction of daily PM2.5 resulted in R2 = 0.86, RMSE = 14.98, and MAPE = 23.72%.The results of this study indicate that the two-step interpolation method can largely resolve the non-random missing data problem and that the combined XGBoost methods have a good ability to estimate fine particulate matter concentrations.

Pragmatic randomized clinical trials: best practices and statistical guidance

Randomized clinical trials often serve the purpose of assessing the efficacy and safety of a compound. By combining real-world evidence and randomization, pragmatic randomized clinical trials (PrCTs) can be used to inform treatment effectiveness and healthcare decisions. PrCTs, referring to studies where several pragmatic elements are used (eligibility, endpoints, follow-up, etc.), pose unique challenges (Loudon et al. in BMJ 350:h2147, 2015). From a literature review, we propose a definition of PrCT and discuss strategies to overcome some PrCT challenges. Use of alternative data collection approaches may lead to uncertainties, and absence of blinding could potentially lead to non-random missing data at study endpoints such that randomization is no longer protected by an intent to treat. Therefore, more complex randomization strategies may be needed to minimize bias. Additional data sources could be used to synthesize information and create a more accurate endpoint definition, which may require tools such as natural language processing. The statistician must become familiar with the challenges and strengths of PrCTs, ranging from design to analysis to interpretation, in order to transform data into evidence (Califf in Clin Trials 13:471–477, 2016).

How differing methods of ascribing ethnicity and socio-economic status affect risk estimates for hospitalisation with infectious disease.

Significant ethnic and socio-economic disparities exist in infectious diseases (IDs) rates in New Zealand, so accurate measures of these characteristics are required. This study compared methods of ascribing ethnicity and socio-economic status. Children in the Growing Up in New Zealand longitudinal cohort were ascribed to self-prioritised, total response and single-combined ethnic groups. Socio-economic status was measured using household income, and both census-derived and survey-derived deprivation indices. Rates of ID hospitalisation were compared using linked administrative data. Self-prioritised ethnicity was simplest to use. Total response accounted for mixed ethnicity and allowed overlap between groups. Single-combined ethnicity required aggregation of small groups to maintain power but offered greater detail. Regardless of the method used, Māori and Pacific children, and children in the most socio-economically deprived households had a greater risk of ID hospitalisation. Risk differences between self-prioritised and total response methods were not significant for Māori and Pacific children but single-combined ethnicity revealed a diversity of risk within these groups. Household income was affected by non-random missing data. The census-derived deprivation index offered a high level of completeness with some risk of multicollinearity and concerns regarding the ecological fallacy. The survey-derived index required extra questions but was acceptable to participants and provided individualised data. Based on these results, the use of single-combined ethnicity and an individualised survey-derived index of deprivation are recommended where sample size and data structure allow it.

A graphical model for skewed matrix-variate non-randomly missing data.

Epidemiological studies on periodontal disease (PD) collect relevant bio-markers, such as the clinical attachment level (CAL) and the probed pocket depth (PPD), at pre-specified tooth sites clustered within a subject's mouth, along with various other demographic and biological risk factors. Routine cross-sectional evaluation are conducted under a linear mixed model (LMM) framework with underlying normality assumptions on the random terms. However, a careful investigation reveals considerable non-normality manifested in those random terms, in the form of skewness and tail behavior. In addition, PD progression is hypothesized to be spatially-referenced, i.e. disease status at proximal tooth-sites may be different from distally located sites, and tooth missingness is non-random (or informative), given that the number and location of missing teeth informs about the periodontal health in that region. To mitigate these complexities, we consider a matrix-variate skew-$t$ formulation of the LMM with a Markov graphical embedding to handle the site-level spatial associations of the bivariate (PPD and CAL) responses. Within the same framework, the non-randomly missing responses are imputed via a latent probit regression of the missingness indicator over the responses. Our hierarchical Bayesian framework powered by relevant Markov chain Monte Carlo steps addresses the aforementioned complexities within an unified paradigm, and estimates model parameters with seamless sharing of information across various stages of the hierarchy. Using both synthetic and real clinical data assessing PD status, we demonstrate a significantly improved fit of our proposition over various other alternative models.