Nonignorable Missing Data Mechanism Research Articles

BAYESIAN MODELING LONGITUDINAL DYADIC DATA WITH NONIGNORABLE DROPOUT, WITH APPLICATION TO A BREAST CANCER STUDY.

Dyadic data are common in the social and behavioral sciences, in which members of dyads are correlated due to the interdependence structure within dyads. The analysis of longitudinal dyadic data becomes complex when nonignorable dropouts occur. We propose a fully Bayesian selection-model-based approach to analyze longitudinal dyadic data with nonignorable dropouts. We model repeated measures on subjects by a transition model and account for within-dyad correlations by random effects. In the model, we allow subject's outcome to depend on his/her own characteristics and measure history, as well as those of the other member in the dyad. We further account for the nonignorable missing data mechanism using a selection model in which the probability of dropout depends on the missing outcome. We propose a Gibbs sampler algorithm to fit the model. Simulation studies show that the proposed method effectively addresses the problem of nonignorable dropouts. We illustrate our methodology using a longitudinal breast cancer study.

Land Cover Classification of Cloud-Contaminated Multitemporal High-Resolution Images

We show how methods proposed in the statistical community dealing with missing data may be applied for land cover classification, where optical observations are missing due to clouds and snow. The proposed method is divided into two stages: 1) cloud/snow classification and 2) training and land cover classification. The purpose of the cloud/snow classification stage is to determine which pixels are missing due to clouds and snow. All pixels in each optical image are classified into the classes cloud, snow, water, and vegetation using a suitable classifier. The pixels classified as cloud or snow are labeled as missing, and this information is used in the subsequent training and classification stage, which deals with classification of the pixels into various land cover classes. For land cover classification, we apply the maximum-likelihood (assuming normal distributions), -nearest neighbor, and Parzen classifiers, all modified to handle missing features. The classifiers are evaluated on Landsat (both Thematic Mapper and Enhanced Thematic Mapper Plus) images covering a scene at about 900 m a.s.l. in the Hardangervidda mountain plateau in Southern Norway, where 4869 in situ samples of the land cover classes water, ridge, leeside, snowbed, mire, forest, and rock are obtained. The results show that proper modeling of the missing pixels improves the classification rate by 5%-10%, and by using multiple images, we increase the chance of observing the land cover type substantially. The nonparametric classifiers handle nonignorable missing-data mechanisms and are therefore particularly suitable for remote sensing applications where the pixels covered by snow and cloud may depend on the land cover type.

Adjusting for Nonignorable Missingness When Estimating Generalized Additive Models

Generalized additive models (GAMs) have been widely used for flexible modeling of various types of outcomes. When the outcome in a GAM is subject to missing, practical analyses often assume that missingness is missing at random (MAR). This assumption can be of suspicion when the missingness is not by design. Evaluating the potential effects of alternative nonignorable missing data mechanism on the MAR inference from a GAM can be important but often challenging due to the complicatedness of alternative nonignorable models. We apply the index approach to local sensitivity (Troxel, Ma, and Heitjan (2004). Statistica Sinica 14, 1221-1237) to evaluate the potential changes of the GAM estimates in the neighborhood of the MAR model. The approach avoids fitting any complicated nonignorable GAM. Only MAR estimates are required to calculate the resulting sensitivity index and adjust the GAM estimates to account for nonignorable missingness. Thus the proposed approach is considerably simpler to conduct, as compared with the alternative methods. The simulation study shows that the index provides valid assessment of the local sensitivity of the GAM estimates to nonignorable missingness. We then illustrate the method using a rheumatoid arthritis clinical trial data set.

Estimation of average treatment effect with incompletely observed longitudinal data: Application to a smoking cessation study

We study the problem of estimation and inference on the average treatment effect in a smoking cessation trial where an outcome and some auxiliary information were measured longitudinally, and both were subject to missing values. Dynamic generalized linear mixed effects models linking the outcome, the auxiliary information, and the covariates are proposed. The maximum likelihood approach is applied to the estimation and inference on the model parameters. The average treatment effect is estimated by the G-computation approach, and the sensitivity of the treatment effect estimate to the nonignorable missing data mechanisms is investigated through the local sensitivity analysis approach. The proposed approach can handle missing data that form arbitrary missing patterns over time. We applied the proposed method to the analysis of the smoking cessation trial.

A Bayesian Model for Longitudinal Count Data with Non-Ignorable Dropout

Asthma is an important chronic disease of childhood. An intervention programme for managing asthma was designed on principles of self-regulation and was evaluated by a randomized longitudinal study.The study focused on several outcomes, and, typically, missing data remained a pervasive problem. We develop a pattern-mixture model to evaluate the outcome of intervention on the number of hospitalizations with non-ignorable dropouts. Pattern-mixture models are not generally identifiable as no data may be available to estimate a number of model parameters. Sensitivity analyses are performed by imposing structures on the unidentified parameters.We propose a parameterization which permits sensitivity analyses on clustered longitudinal count data that have missing values due to non-ignorable missing data mechanisms. This parameterization is expressed as ratios between event rates across missing data patterns and the observed data pattern and thus measures departures from an ignorable missing data mechanism. Sensitivity analyses are performed within a Bayesian framework by averaging over different prior distributions on the event ratios. This model has the advantage of providing an intuitive and flexible framework for incorporating the uncertainty of the missing data mechanism in the final analysis.

Data pruning in consumer choice models

Common, if not ubiquitous, Marketing practice when estimating models for scanner panel data is to: (a) observe the data, (b) prune the data to a “manageable” number of brands or SKUs, and (c) fit models to the remaining data. We demonstrate that such pruning practice can lead to significantly different (and potentially biased) elasticities, and hence different managerial/practical outcomes, especially in the context of model misspecification. We first justify our claims theoretically by writing the general problem in a classic missing-data framework and demonstrate that commonly used pruning mechanisms (gleaned from current academic Marketing literature) can lead to a nonignorable missing data mechanism. Secondly, we summarize an extensive set of simulations that were run to understand the driving factors of that bias. The results indicate much greater pruning bias in those cases where model fit is poor (small \(R^{2}\)), random utility errors are correlated with the covariates, or the model is misspecified (e.g., a homogeneous logit is specified when a mixed-logit is true). Empirically, we also demonstrate our findings on the well-cited and highly utilized fabric softener data of Fader and Hardie (1996). Our empirical findings suggest a number of estimates that vary according to the way in which the data is pruned including the magnitude of market mix and attribute elasticities, and purchase probabilities, but that the pruning effect is smaller for better fitting models.

A Bayesian Approach for Clustered Longitudinal Ordinal Outcome With Nonignorable Missing Data

Asthma, a chronic inflammatory disease of the airways, affects an estimated 6.3 million children under age 18 in the United States. A key to successful asthma management, and hence improved quality of life (QOL), calls for an active partnership between asthma patients and their health care providers. To foster this partnership, an intervention program was designed and evaluated using a randomized longitudinal study. The study focused on several outcomes where typically missing data remained a pervasive problem. We suspected that the underlying missing-data mechanism may not be ignorable. Thus here we present a method for analyzing clustered longitudinal data with missing values resulting from a nonignorable missing-data mechanism. The transition Markov model with random effects was used to investigate changes in ordinal outcomes over time. A Bayesian pattern-mixture model with the flexibility to incorporate models for missing data in both outcome and time-varying covariates was used to model the nonignorable missing-data mechanism. The pattern-mixture model uses easy-to-understand parameters—namely, ratios of the cumulative odds across patterns with the complete-data pattern—as the reference pattern. Sensitivity analysis was performed using different prior distributions for the parameters. A fully Bayesian approach was derived by integrating over a class of prior distributions. The data from the Asthma Intervention Study were analyzed to explore the effect of the intervention program on improving QOL.

Using Data Augmentation to Correct for Non-Ignorable Non-Response When Surrogate Data are Available: An Application to the Distribution of Hourly Pay

SummaryThe paper develops a data augmentation method to estimate the distribution function of a variable, which is partially observed, under a non-ignorable missing data mechanism, and where surrogate data are available. An application to the estimation of hourly pay distributions using UK Labour Force Survey data provides the main motivation. In addition to considering a standard parametric data augmentation method, we consider the use of hot deck imputation methods as part of the data augmentation procedure to improve the robustness of the method. The method proposed is compared with standard methods that are based on an ignorable missing data mechanism, both in a simulation study and in the Labour Force Survey application. The focus is on reducing bias in point estimation, but variance estimation using multiple imputation is also considered briefly.

Modelling non-ignorable missing-data mechanisms with item response theory models

A model-based procedure for assessing the extent to which missing data can be ignored and handling non-ignorable missing data is presented. The procedure is based on item response theory modelling. As an example, the approach is worked out in detail in conjunction with item response data modelled using the partial credit and generalized partial credit models. Simulation studies are carried out to assess the extent to which the bias caused by ignoring the missing-data mechanism can be reduced. Finally, the feasibility of the procedure is demonstrated using data from a study to calibrate a medical disability scale.

Asymptotic Bias in the Linear Mixed Effects Model Under Non-Ignorable Missing Data Mechanisms

SummaryIn longitudinal studies, missingness of data is often an unavoidable problem. Estimators from the linear mixed effects model assume that missing data are missing at random. However, estimators are biased when this assumption is not met. In the paper, theoretical results for the asymptotic bias are established under non-ignorable drop-out, drop-in and other missing data patterns. The asymptotic bias is large when the drop-out subjects have only one or no observation, especially for slope-related parameters of the linear mixed effects model. In the drop-in case, intercept-related parameter estimators show substantial asymptotic bias when subjects enter late in the study. Eight other missing data patterns are considered and these produce asymptotic biases of a variety of magnitudes.

Protective estimator for linear regression with nonignorably missing Gaussian outcomes

We propose a method for estimating the regression parameters in a linear regression model for Gaussian data when the outcome variable is missing for some subjects and missingness is thought to be nonignorable. Throughout, we assume that missingness is restricted to the outcome variable and that the covariates are fully observed. Although maximum likelihood estimation of the regression parameters is possible once joint models for the outcome variable and the nonignorable missing data mechanism have been specified, these models are fundamentally nonidentifiable unless unverifiable modeling assumptions are imposed. In this paper, rather than explicitly modeling the nonignorable missingness mechanism, we consider the use of a ‘protective’ estimator of the regression parameters (Brown, 1990). To implement the proposed method, it is necessary to assume that the outcome variable and one of the covariates have an approximate bivariate normal distribution, conditional on the remaining covariates. In addition, it is assumed that the missing data mechanism is conditionally independent of this covariate, given the outcome variable and the remaining covariates; the latter is referred to as the ‘protective’ assumption. A method of moments approach is used to obtain the protective estimator of the regression parameters; the jackknife (Quenouille, 1956) is used to estimate the variance. The method is illustrated using data on the persistence of maternal smoking from the Six Cities Study of the health effects of air pollution (Ware et al., 1984). The results of a simulation study are presented that examine the magnitude of any finite sample bias.

A global sensitivity analysis of performance of a medical diagnostic test when verification bias is present.

Current advances in technology provide less invasive or less expensive diagnostic tests for identifying disease status. When a diagnostic test is evaluated against an invasive or expensive gold standard test, one often finds that not all patients undergo the gold standard test. The sensitivity and specificity estimates based only on the patients with verified disease are often biased. This bias is called verification bias. Many authors have examined the consequences of verification bias and have proposed bias correction methods based on the assumption of independence between disease status and election for verification conditionally on the test result, or equivalently on the assumption that the disease status is missing at random using missing data terminology. This assumption may not be valid and one may need to consider adjustment for a possible non-ignorable verification bias resulting from the non-ignorable missing data mechanism. Such an adjustment involves ultimately uncheckable assumptions and requires sensitivity analysis. The sensitivity analysis is most often accomplished by perturbing parameters in the chosen model for the missing data mechanism, and it has a local flavour because perturbations are around the fitted model. In this paper we propose a global sensitivity analysis for assessing performance of a diagnostic test in the presence of verification bias. We derive a region of all sensitivity and specificity values consistent with the observed data and call this region a test ignorance region (TIR). The term 'ignorance' refers to the lack of knowledge due to the missing disease status for the not verified patients. The methodology is illustrated with two clinical examples.

Examining the Impact of Missing Data on Propensity Score Estimation in Determining the Effectiveness of Self-Monitoring of Blood Glucose (SMBG)

In many health services applications, research to determine the effectiveness of a particular treatment cannot be carried out using a controlled clinical trial. In settings such as these, observational studies must be used. Propensity score methods are useful tools to employ in order to balance the distribution of covariates between treatment groups and hence reduce the potential bias in treatment effect estimates in observational studies. A challenge in many health services research studies is the presence of missing data among the covariates that need to be balanced. In this paper, we compare three simple propensity models using data that examine the effectiveness of self-monitoring of blood glucose (SMBG) in reducing hemoglobin A1c in a cohort of 10,566 type 2 diabetics. The first propensity score model uses only subjects with complete case data (n=6,687), the second incorporates missing value indicators into the model, and the third fits separate propensity scores for each pattern of missing data. We compare the results of these methods and find that incorporating missing data into the propensity score model reduces the estimated effect of SMBG on hemoglobin A1c by more than 10%, although this reduction was not clinically significant. In addition, beginning with the complete data, we artificially introduce missing data using a nonignorable missing data mechanism and compare treatment effect estimates using the three propensity score methods and a simple analysis of covariance (ANCOVA) method. In these analyses, we find that the complete case analysis and the ANCOVA method both perform poorly, the missing value indicator model performs moderately well, and the pattern mixture model performs even better in estimating the original treatment effect observed in the␣complete data prior to the introduction of artificial missing data. We conclude that in observational studies one␣must not only adjust for potentially confounding variables using methods such as propensity scores, but one␣should also account for missing data in these models in order to allow for causal inference more appropriately to be applied.

Reparameterizing the pattern mixture model for sensitivity analyses under informative dropout.

Pattern mixture models are frequently used to analyze longitudinal data where missingness is induced by dropout. For measured responses, it is typical to model the complete data as a mixture of multivariate normal distributions, where mixing is done over the dropout distribution. Fully parameterized pattern mixture models are not identified by incomplete data; Little (1993, Journal of the American Statistical Association 88, 125-134) has characterized several identifying restrictions that can be used for model fitting. We propose a reparameterization of the pattern mixture model that allows investigation of sensitivity to assumptions about nonidentified parameters in both the mean and variance, allows consideration of a wide range of nonignorable missing-data mechanisms, and has intuitive appeal for eliciting plausible missing-data mechanisms. The parameterization makes clear an advantage of pattern mixture models over parametric selection models, namely that the missing-data mechanism can be varied without affecting the marginal distribution of the observed data. To illustrate the utility of the new parameterization, we analyze data from a recent clinical trial of growth hormone for maintaining muscle strength in the elderly. Dropout occurs at a high rate and is potentially informative. We undertake a detailed sensitivity analysis to understand the impact of missing-data assumptions on the inference about the effects of growth hormone on muscle strength.

Non-ignorable missing covariates in generalized linear models

We propose a likelihood method for estimating parameters in generalized linear models with missing covariates and a non-ignorable missing data mechanism. In this paper, we focus on one missing covariate. We use a logistic model for the probability that the covariate is missing, and allow this probability to depend on the incomplete covariate. We allow the covariates, including the incomplete covariate, to be either categorical or continuous. We propose an EM algorithm in this case. For a missing categorical covariate, we derive a closed form expression for the E- and M-steps of the EM algorithm for obtaining the maximum likelihood estimates (MLEs). For a missing continuous covariate, we use a Monte Carlo version of the EM algorithm to obtain the MLEs via the Gibbs sampler. The methodology is illustrated using an example from a breast cancer clinical trial in which time to disease progression is the outcome, and the incomplete covariate is a quality of life physical well-being score taken after the start of therapy. This score may be missing because the patients are sicker, so this covariate could be non-ignorably missing.

Missing Covariates in Generalized Linear Models When the Missing Data Mechanism is Non-ignorable

Summary We propose a method for estimating parameters in generalized linear models with missing covariates and a non-ignorable missing data mechanism. We use a multinomial model for the missing data indicators and propose a joint distribution for them which can be written as a sequence of one-dimensional conditional distributions, with each one-dimensional conditional distribution consisting of a logistic regression. We allow the covariates to be either categorical or continuous. The joint covariate distribution is also modelled via a sequence of one-dimensional conditional distributions, and the response variable is assumed to be completely observed. We derive the E- and M-steps of the EM algorithm with non-ignorable missing covariate data. For categorical covariates, we derive a closed form expression for the E- and M-steps of the EM algorithm for obtaining the maximum likelihood estimates (MLEs). For continuous covariates, we use a Monte Carlo version of the EM algorithm to obtain the MLEs via the Gibbs sampler. Computational techniques for Gibbs sampling are proposed and implemented. The parametric form of the assumed missing data mechanism itself is not ‘testable' from the data, and thus the non-ignorable modelling considered here can be viewed as a sensitivity analysis concerning a more complicated model. Therefore, although a model may have ‘passed' the tests for a certain missing data mechanism, this does not mean that we have captured, even approximately, the correct missing data mechanism. Hence, model checking for the missing data mechanism and sensitivity analyses play an important role in this problem and are discussed in detail. Several simulations are given to demonstrate the methodology. In addition, a real data set from a melanoma cancer clinical trial is presented to illustrate the methods proposed.

Effect of non‐random missing data mechanisms in clinical trials

A simple form of non-ignorable missing data mechanisms based on two parameters is used to characterize the amount of missing data and the severity of non-randomness in clinical trials. Based on the formulation, the effect of non-randomly missing data on simple analyses which ignore the missing data is studied for binary and normally distributed response variables. In general, the effect of the non-randomly missing data on the bias and the power increases with the severity of non-randomness. The bias can be positive or negative and the power can be less than or greater than when the data are missing at random. The results of the analysis, ignoring the missing data, can be seriously flawed if the non-randomness is severe, even when only a small proportion of the sample is missing. The problem is more pronounced in the case of normally distributed response variables with unequal variances.