Nonignorable Missing Data Research Articles

Rapid earthquake impact prediction using non-ignorable missing data from non-expert observers

A rapid method for predicting earthquake impacts is proposed that uses spatial data received from non-expert individuals immediately after an earthquake. The approach addresses the issue of non-ignorable missing data caused by spatial latent random fields, which seriously affect the accuracy of predictions. A spatial model was employed to jointly model missingness and earthquake impact measurement by considering the latent spatial random fields as shared parameters. The proposed method demonstrated better prediction compared with that of existing models in both simulated and actual datasets. When applied to the real-world 2017 Sarpol-e Zahab earthquake in Iran, the model revealed concentrated impacts south and southeast of the epicenter.

Direct Estimation for Commonly Used Pattern-Mixture Models in Clinical Trials

Pattern-mixture models have received increasing attention as they are commonly used to assess treatment effects in primary or sensitivity analyses for clinical trials with nonignorable missing data. Pattern-mixture models have traditionally been implemented using multiple imputation, where the variance estimation may be a challenge because the Rubin’s approach of combining between- and within-imputation variance may not provide consistent variance estimation while bootstrap methods may be time-consuming. Direct likelihood-based approaches have been proposed in the literature and implemented for some pattern-mixture models, but the assumptions are sometimes restrictive and the theoretical framework is fragile. In this article, we propose an analytical framework for an efficient direct likelihood estimation method for commonly used pattern-mixture models corresponding to return-to-baseline, jump-to-reference, placebo washout, and retrieved dropout imputations. A parsimonious tipping point analysis is also discussed and implemented. Results from simulation studies demonstrate that the proposed methods provide consistent estimators. We further illustrate the utility of the proposed methods using data from a clinical trial evaluating a treatment for type 2 diabetes.

Quantile regression and smoothed empirical likelihood for non-ignorable missing data based on semi-parametric response models

Quantile regression and smoothed empirical likelihood for non-ignorable missing data based on semi-parametric response models

Identification and Semiparametric Efficiency Theory of Nonignorable Missing Data with a Shadow Variable

We consider identification and estimation with an outcome missing not at random (MNAR). We study an identification strategy based on a so-called shadow variable . A shadow variable is assumed to be correlated with the outcome but independent of the missingness process conditional on the outcome and fully observed covariates. We describe a general condition for nonparametric identification of the full data law under MNAR using a valid shadow variable. Our condition is satisfied by many commonly used models; moreover, it is imposed on the complete cases, and therefore has testable implications with observed data only. We characterize the semiparametric efficiency bound for the class of regular and asymptotically linear estimators and derive a closed form for the efficient influence function. We describe a doubly robust and locally efficient estimation method and evaluate its performance on both simulation data and a real data example about home pricing.

A mixed model approach to estimate the survivor average causal effect in cluster-randomized trials.

In many medical studies, the outcome measure (such as quality of life, QOL) for some study participants becomes informatively truncated (censored, missing, or unobserved) due to death or other forms of dropout, creating a nonignorable missing data problem. In such cases, the use of a composite outcome or imputation methods that fill in unmeasurable QOL values for those who died rely on strong and untestable assumptions and may be conceptually unappealing to certain stakeholders when estimating a treatment effect. The survivor average causal effect (SACE) is an alternative causal estimand that surmounts some of these issues. While principal stratification has been applied to estimate the SACE in individually randomized trials, methods for estimating the SACE in cluster-randomized trials are currently limited. To address this gap, we develop a mixed model approach along with an expectation-maximization algorithm to estimate the SACE in cluster-randomized trials. We model the continuous outcome measure with a random intercept to account for intracluster correlations due to cluster-level randomization, and model the principal strata membership both with and without a random intercept. In simulations, we compare the performance of our approaches with an existing fixed-effects approach to illustrate the importance of accounting for clustering in cluster-randomized trials. The methodology is then illustrated using a cluster-randomized trial of telecare and assistive technology on health-related QOL in the elderly.

Variable selection for semiparametric accelerated failure time models with nonignorable missing data

Variable selection for semiparametric accelerated failure time models with nonignorable missing data

Bayesian Ideas in Survey Sampling: The Legacy of Basu

Survey sampling and, more generally, Official Statistics are experiencing an important renovation time. On one hand, there is the need to exploit the huge information potentiality that the digital revolution made available in terms of data. On the other hand, this process occurred simultaneously with a progressive deterioration of the quality of classical sample surveys, due to a decreasing willingness to participate and an increasing rate of missing responses. The switch from survey-based inference to a hybrid system involving register-based information has made more stringent the debate and the possible resolution of the design-based versus model-based approaches controversy. In this new framework, the use of statistical models seems unavoidable and it is today a relevant part of the official statistician toolkit. Models are important in several different contexts, from Small area estimation to non sampling error adjustment, but they are also crucial for correcting bias due to over and undercoverage of administrative data, in order to prevent potential selection bias, and to deal with different definitions and/or errors in the measurement process of the administrative sources. The progressive shift from a design-based to a model-based approach in terms of super-population is a matter of fact in the practice of the National Statistical Institutes. However, the introduction of Bayesian ideas in official statistics still encounters difficulties and resistance. In this work, we attempt a non-systematic review of the Bayesian development in this area and try to highlight the extra benefit that a Bayesian approach might provide. Our general conclusion is that, while the general picture is today clear and most of the basic topics of survey sampling can be easily rephrased and tackled from a Bayesian perspective, much work is still necessary for the availability of a ready-to-use platform of Bayesian survey sampling in the presence of complex sampling design, non-ignorable missing data patterns, and large datasets.

Assessable and interpretable sensitivity analysis in the pattern graph framework for nonignorable missingness mechanisms.

The pattern graph framework solves a wide range of missing data problems with nonignorable mechanisms. However, it faces two challenges of assessability and interpretability, particularly important in safety-critical problems such as clinical diagnosis: (i) How can one assess the validity of the framework's apriori assumption and make necessary adjustments to accommodate known information about the problem? (ii) How can one interpret the process of exponential tilting used for sensitivity analysis in the pattern graph framework and choose the tilt perturbations based on meaningful real-world quantities? In this paper, we introduce Informed Sensitivity Analysis, an extension of the pattern graph framework that enables us to incorporate substantive knowledge about the missingness mechanism into the pattern graph framework. Our extension allows us to examine the validity of assumptions underlying pattern graphs and interpret sensitivity analysis results in terms of realistic problem characteristics. We apply our method to a prevalent nonignorable missing data scenario in clinical research. We validate and compare our method'sresults of our method with a number of widely-used missing data methods, including Unweighted CCA, KNN Imputer, MICE, and MissForest. The validation is done using both boot-strapped simulated experiments as well as real-world clinical observations in the MIMIC-III public dataset.

Semiparametric multivariate joint model for skewed-longitudinal and survival data: A Bayesian approach.

Joint models and statistical inference for longitudinal and survival data have been an active area of statistical research and have mostly coupled a longitudinal biomarker-based mixed-effects model with normal distribution and an event time-based survival model. In practice, however, the following issues may standout: (i) Normality of model error in longitudinal models is a routine assumption, but it may be unrealistically violating data features of subject variations. (ii) Data collected are often featured by the mixed types of multiple longitudinal outcomes which are significantly correlated, ignoring their correlation may lead to biased estimation. Additionally, a parametric model specification may be inflexible to capture the complicated patterns of longitudinal data. (iii) Missing observations in the longitudinal data are often encountered; the missing measures are likely to be informative (nonignorable) and ignoring this phenomenon may result in inaccurate inference. Multilevel item response theory (MLIRT) models have been increasingly used to analyze the multiple longitudinal data of mixed types (ie, continuous and categorical) in clinical studies. In this article, we develop an MLIRT-based semiparametric joint model with skew-t distribution that consists of an extended MLIRT model for the mixed types of multiple longitudinal data and a Cox proportional hazards model, linked through random-effects. A Bayesian approach is employed for joint modeling. Simulation studies are conducted to assess performance of the proposed models and method. A real example from primary biliary cirrhosis clinical study is analyzed to estimate parameters in the joint model and also evaluate sensitivity of parameter estimates for various plausible nonignorable missing data mechanisms.

A self-censoring model for multivariate nonignorable nonmonotone missing data.

We introduce an itemwise modeling approach called "self-censoring" for multivariate nonignorable nonmonotone missing data, where the missingness process of each outcome can be affected by its own value and associated with missingness indicators of other outcomes, while conditionally independent of the other outcomes. The self-censoring model complements previous graphical approaches for the analysis of multivariate nonignorable missing data. It is identified under a completeness condition stating that any variability in one outcome can be captured by variability in the other outcomes among complete cases. For estimation, we propose a suite of semiparametric estimators including doubly robust estimators that deliver valid inferences under partial misspecification of the full-data distribution. We also provide a novel and flexible global sensitivity analysis procedure anchored at the self-censoring. We evaluate the performance of the proposed methods with simulations and apply them to analyze a study about the effect of highly active antiretroviral therapy on preterm delivery of HIV-positivemothers.

Instability of inverse probability weighting methods and a remedy for nonignorable missing data.

Inverse probability weighting (IPW) methods are commonly used to analyze nonignorable missing data (NIMD) under the assumption of a logistic model for the missingness probability. However, solving IPW equationsnumerically may involve nonconvergence problems when the sample size is moderate and the missingness probability is high. Moreover, those equationsoften have multiple roots, and identifying the best root is challenging. Therefore, IPW methods may have low efficiency or even produce biased results. We identify the pitfall in these methods pathologically: they involve the estimation of a moment-generating function (MGF), and such functions are notoriously unstable in general. As a remedy, we model the outcome distribution given the covariates of the completely observed individuals semiparametrically. After forming an induced logistic regression (LR) model for the missingness status of the outcome and covariate, we develop a maximum conditional likelihood method to estimate the underlying parameters. The proposed method circumvents the estimation of an MGF and hence overcomes the instability of IPW methods. Our theoretical and simulation results show that the proposed method outperforms existing competitors greatly. Two real data examples are analyzed to illustrate the advantages of our method. We conclude that if only a parametric LR is assumed but the outcome regression model is left arbitrary, then one has to be cautious in using any of the existing statistical methods in problems involvingNIMD.

A Bayesian feature allocation model for identifying cell subpopulations using CyTOF data.

A Bayesian feature allocation model (FAM) is presented for identifying cell subpopulations based on multiple samples of cell surface or intracellular marker expression level data obtained by cytometry by time of flight (CyTOF). Cell subpopulations are characterized by differences in marker expression patterns, and cells are clustered into subpopulations based on their observed expression levels. A model-based method is used to construct cell clusters within each sample by modeling subpopulations as latent features, using a finite Indian buffet process. Non-ignorable missing data due to technical artifacts in mass cytometry instruments are accounted for by defining a static missingship mechanism. In contrast with conventional cell clustering methods, which cluster observed marker expression levels separately for each sample, the FAM-based method can be applied simultaneously to multiple samples, and also identify important cell subpopulations likely to be otherwise missed. The proposed FAM-based method is applied to jointly analyse three CyTOF datasets to study natural killer (NK) cells. Because the subpopulations identified by the FAM may define novel NK cell subsets, this statistical analysis may provide useful information about the biology of NK cells and their potential role in cancer immunotherapy which may lead, in turn, to development of improved NK cell therapies.

Fine-tuned sensitivity analysis for non-ignorable missing data mechanism in linear regression models

Fine-tuned sensitivity analysis for non-ignorable missing data mechanism in linear regression models

Bayesian joint modeling of binomial and rank response with non-ignorable missing data for primate cognition

A random effects model for analyzing mixed rank and binomial data with considering the missing values is presented. Occurring of missing data is an important problem in all research fields. The most common approach to dealing with missing data is to delete cases containing missing observations. However, this approach reduces statistical power and mislead us to biased statistical results. This paper aims to prepare guidance for researchers facing missing data problems and to provide techniques for jointly modeling of binomial and rank responses. We compare the cognitive abilities of different primates based on their performance on 17 cognitive assessments obtained on either a rank or binomial scale using Bayesian latent variable with random effects models. Random effects are used to take into account the correlation between responses of the same individual.

Learning and total evidence with imprecise probabilities

In dynamic learning, a rational agent must revise their credence about a question of interest in accordance with the total evidence available between the earlier and later times. We discuss situations in which an observable event F that is sufficient for the total evidence can be identified, yet its probabilistic modeling cannot be performed in a precise manner. The agent may employ imprecise (IP) models of reasoning to account for the identified sufficient event, and perform change of credence or sequential decisions accordingly. Our proposal is illustrated with four case studies: the classic Monty Hall problem, statistical inference with non-ignorable missing data, frequentist hypothesis testing, and the use of forward induction in a two-person sequential game.

Bayesian semiparametric approach to quantile nonlinear dynamic factor analysis models with mixed ordered and nonignorable missing data

In classical nonlinear dynamic factor analysis models (NDFAMs), we assume manifest variables follow the normal distribution. However, in some applications, the normality assumption may be unreasonable or questionable. This paper proposes a quantile NDFAM in which manifest variables may be observed or unobserved but associated with the observed ordinal indicators or subject to missingness, and develops a Bayesian semiparametric method to estimate unknown parameters and latent variables by utilizing the Dirichlet process prior to approximate unknown distribution of random effects. Logistic and probit models are adopted to specify propensity scores for missing manifests and covariates, respectively. An exponential family distribution is assumed for missing covariates. Bayesian local influence is presented to assess the effect of minor perturbations to data, priors, and sampling distributions. Several simulation studies and a real example are illustrated by the proposed methodologies.

Quantile regression for nonignorable missing data with its application of analyzing electronic medical records.

Over the past decade, there has been growing enthusiasm for using electronic medical records (EMRs) for biomedical research. Quantile regression estimates distributional associations, providing unique insights into the intricacies and heterogeneity of the EMR data. However, the widespread nonignorable missing observations in EMR often obscure the true associations and challenge its potential for robust biomedical discoveries. We propose a novel method to estimate the covariate effects in the presence of nonignorable missing responses under quantile regression. This method imposes no parametric specifications on response distributions, which subtly uses implicit distributions induced by the corresponding quantile regression models. We show that the proposed estimator is consistent and asymptotically normal. We also provide an efficient algorithm to obtain the proposed estimate and a randomly weighted bootstrap approach for statistical inferences. Numerical studies, including an empirical analysis of real-world EMR data, are used to assess the proposed method's finite-sample performance compared to existingliterature.

Measuring Individual Benefits of Medical Treatments Using Longitudinal Hospital Data with Non-Ignorable Missing Responses Caused by Patient Discharge: Application to the Study of Benefits of Pain Management Post Spinal Fusion

Electronic health records (EHR) provide valuable resources for longitudinal studies and understanding risk factors associated with poor clinical outcomes. However, they may not contain complete follow-ups, and the missing data may not be at random since hospital discharge may depend in part on expected but unrecorded clinical outcomes that occur after patient discharge. These non-ignorable missing data requires appropriate analysis methods. Here, we are interested in measuring and analyzing individual treatment benefits of medical treatments in patients recorded in EHR databases. We present a method for predicting individual benefits that handles non-ignorable missingness due to hospital discharge. The longitudinal clinical outcome of interest is modeled simultaneously with the hospital length of stay using a joint mixed-effects model, and individual benefits are predicted through a frequentist approach: the empirical Bayesian approach. We illustrate our approach by assessing individual pain management benefits to patients who underwent spinal fusion surgery. By calculating sample percentiles of empirical Bayes predictors of individual benefits, we examine the evolution of individual benefits over time. We additionally compare these percentiles with percentiles calculated with a Monte Carlo approach. We showed that empirical Bayes predictors of individual benefits do not only allow examining benefits in specific patients but also reflect overall population trends reliably.

Model parameters estimation with non-ignorable missing data using influential exponential tilting resampling approach

This paper proposes to extend [1] mean functional estimation method based on the influential exponential tilting resampling approach (ITRA) to address non-ignorable missing data in linear model parameters statistical inference. The ITRA approach assumes that the nonrespondents’ model corresponds to an exponential tilting of the respondents’ model. The tilted model's specified function is the influential function of the function of interest (parameter). The other basis of the proposed approach is to use the importance resampling techniques to draw inferences about some linear model parameters. Simulation studies were conducted to investigate the performance of the proposed methods and their application to real data. Theoretical justifications are provided as well.

Mean functional estimation with non-ignorable missing data using influential exponential tilting resampling approach

Inference with incomplete data, especially non-ignorable, is one of the challenging tasks in data analysis. Moreover, it is well known that results obtained from incomplete data are prone to bias. Parameter estimation with non-ignorable missing data is even more challenging to handle and extract useful information. This paper proposes a method based on the Influential tilting resampling approach (ITRA) to handle non-ignorable missing data for inference on the mean. One of the bases of the proposed approach is assuming that the model for the non-respondents corresponds to an exponential tilting of the respondents’ model, and the specified function in the tilted model is the influential function of the functional (parameter) of interest. Extensive simulation studies were conducted to investigate the performance of the proposed methods. We provided the theoretical justification, as well as application to real data.