Confidence Interval Coverage Research Articles

Adjusting for partial invariance in latent parameter estimation: Comparing forward specification search and approximate invariance methods.

Measurement invariance is the condition that an instrument measures a target construct in the same way across subgroups, settings, and time. In psychological measurement, usually only partial, but not full, invariance is achieved, which potentially biases subsequent parameter estimations and statistical inferences. Although existing literature shows that a correctly specified partial invariance model can remove such biases, it ignores the model uncertainty in the specification search step: flagging the wrong items may lead to additional bias and variability in subsequent inferences. On the other hand, several new approaches, including Bayesian approximate invariance and alignment optimization methods, have been proposed; these methods use an approximate invariance model to adjust for partial measurement invariance without the need to directly identify noninvariant items. However, there has been limited research on these methods in situations with a small number of groups. In this paper, we conducted three systematic simulation studies to compare five methods for adjusting partial invariance. While specification search performed reasonably well when the proportion of noninvariant parameters was no more than one-third, alignment optimization overall performed best across conditions in terms of efficiency of parameter estimates, confidence interval coverage, and type I error rates. In addition, the Bayesian version of alignment optimization performed best for estimating latent means and variances in small-sample and low-reliability conditions. We thus recommend the use of the alignment optimization methods for adjusting partial invariance when comparing latent constructs across a few groups.

Conducting density-sampled case-control studies using survey data with complex sampling designs: A simulation study

PurposePopulation-based surveys are possible sources from which to draw representative control data for case-control studies. However, these surveys involve complex sampling that could lead to biased estimates of measures of association if not properly accounted for in analyses. Approaches to incorporating complex-sampled controls in density-sampled case-control designs have not been examined. MethodsWe used a simulation study to evaluate the performance of different approaches to estimating incidence density ratios (IDR) from case-control studies with controls drawn from complex survey data using risk-set sampling. In simulated population data, we applied four survey sampling approaches, with varying survey sizes, and assessed the performance of four analysis methods for incorporating survey-based controls. ResultsEstimates of the IDR were unbiased for methods that conducted risk-set sampling with probability of selection proportional to survey weights. Estimates of the IDR were biased when sampling weights were not incorporated, or only included in regression modeling. The unbiased analysis methods performed comparably and produced estimates with variance comparable to biased methods. Variance increased and confidence interval coverage decreased as survey size decreased. ConclusionsUnbiased estimates are obtainable in risk-set sampled case-control studies using controls drawn from complex survey data when weights are properly incorporated.

Abstract 411: New estimators for mediation effects in complex sampling schemes: A comparison with applications

Abstract Mediation analysis examines the indirect effects via a third variable on the pathway between exposures and outcomes. For the finite-population surveys, that often use stratified cluster sampling, the conventional methods assuming simple random sampling are not appropriate. The point estimation of the effect can be corrected using sampling weights, but the standard error needs further improvement. Recent studies have integrated mediation analysis with balanced repeated replication for standard error estimation. Alternative estimators include Jackknife repeated replication (JRR), bootstrap, and Taylor series linear approximation (TSE). This study proposes new standard error estimators by incorporating the classic linear mediation analysis with JRR, bootstrap, and TSE for complex surveys and compares them using Monte Carlo simulations. The preliminary results show that replication methods have lower Type-I error rates and higher confidence-interval coverage rates than conventional methods. The proposed methods are applied to National Health Interview Surveys to explore the mediation effects on the pathway between cancer survivors' health status and employment status. Citation Format: Yujiao Mai, Deo Kumar Srivastava. New estimators for mediation effects in complex sampling schemes: A comparison with applications [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr 411.

Bayesian Inference for the Parameters of Kumaraswamy Distribution via Ranked Set Sampling

In this paper, we address the estimation of the parameters for a two-parameter Kumaraswamy distribution by using the maximum likelihood and Bayesian methods based on simple random sampling, ranked set sampling, and maximum ranked set sampling with unequal samples. The Bayes loss functions used are symmetric and asymmetric. The Metropolis-Hastings-within-Gibbs algorithm was employed to calculate the Bayes point estimates and credible intervals. We illustrate a simulation experiment to compare the implications of the proposed point estimators in sense of bias, estimated risk, and relative efficiency as well as evaluate the interval estimators in terms of average confidence interval length and coverage percentage. Finally, a real-life example and remarks are presented.

Nonlinear mixed‐effects modeling of longitudinal count data: Bayesian inference about median counts based on the marginal zero‐inflated discrete Weibull distribution

This article proposes a Bayesian regression model for nonlinear zero-inflated longitudinal count data that models the median count as an alternative to the mean count. The nonlinear model generalizes a recently introduced linear mixed-effects model based on the zero-inflated discrete Weibull (ZIDW) distribution. The ZIDW distribution is more robust to severe skewness in the data than conventional zero-inflated count distributions such as the zero-inflated negative binomial (ZINB) distribution. Moreover, the ZIDW distribution is attractive because of its convenience to model the median counts given its closed-form quantile function. The median is a more robust measure of central tendency than the mean when the data, for instance, zero-inflated counts, are right-skewed. In an application of the model we consider a biphasic mixed-effects model consisting of an intercept term and two slope terms. Conventionally, the ZIDW model separately specifies the predictors for the zero-inflation probability and the counting process's median count. In our application, the two latent class interpretations are not clinically plausible. Therefore, we propose a marginal ZIDW model that directly models the biphasic median counts marginally. We also consider the marginal ZINB model to make inferences about the nonlinear mean counts over time. Our simulation study shows that the models have good properties in terms of accuracy and confidence interval coverage.

Fisher transformation based confidence intervals of correlations in fixed- and random-effects meta-analysis.

Meta-analyses of correlation coefficients are an important technique to integrate results from many cross-sectional and longitudinal research designs. Uncertainty in pooled estimates is typically assessed with the help of confidence intervals, which can double as hypothesis tests for two-sided hypotheses about the underlying correlation. A standard approach to construct confidence intervals for the main effect is the Hedges-Olkin-Vevea Fisher-z (HOVz) approach, which is based on the Fisher-z transformation. Results from previous studies (Field, 2005, Psychol. Meth., 10, 444; Hafdahl and Williams, 2009, Psychol. Meth., 14, 24), however, indicate that in random-effects models the performance of the HOVz confidence interval can be unsatisfactory. To this end, we propose improvements of the HOVz approach, which are based on enhanced variance estimators for the main effect estimate. In order to study the coverage of the new confidence intervals in both fixed- and random-effects meta-analysis models, we perform an extensive simulation study, comparing them to established approaches. Data were generated via a truncated normal and beta distribution model. The results show that our newly proposed confidence intervals based on a Knapp-Hartung-type variance estimator or robust heteroscedasticity consistent sandwich estimators in combination with the integral z-to-r transformation (Hafdahl, 2009, Br. J. Math. Stat. Psychol., 62, 233) provide more accurate coverage than existing approaches in most scenarios, especially in the more appropriate beta distribution simulation model.

Understanding the impact of correlation within pair-bonds on Cormack-Jolly-Seber models.

The Cormack–Jolly–Seber (CJS) model and its extensions have been widely applied to the study of animal survival rates in open populations. The model assumes that individuals within the population of interest have independent fates. It is, however, highly unlikely that a pair of animals which have formed a long‐term pairing have dissociated fates.We examine a model extension which allows animals who have formed a pair‐bond to have correlated survival and recapture fates. Using the proposed extension to generate data, we conduct a simulation study exploring the impact that correlated fate data has on inference from the CJS model. We compute Monte Carlo estimates for the bias, range, and standard errors of the parameters of the CJS model for data with varying degrees of survival correlation between mates. Furthermore, we study the likelihood ratio test of sex effects within the CJS model by simulating densities of the deviance. Finally, we estimate the variance inflation factor c^ for CJS models that incorporate sex‐specific heterogeneity.Our study shows that correlated fates between mated animals may result in underestimated standard errors for parsimonious models, significantly deflated likelihood ratio test statistics, and underestimated values of c^ for models taking sex‐specific effects into account.Underestimated standard errors can result in lowered coverage of confidence intervals. Moreover, deflated test statistics will provide overly conservative test results. Finally, underestimated variance inflation factors can lead researchers to make incorrect conclusions about the level of extra‐binomial variation present in their data.

Accounting for confounding by time, early intervention adoption, and time-varying effect modification in the design and analysis of stepped-wedge designs: application to a proposed study design to reduce opioid-related mortality

BackgroundBeginning in 2019, stepped-wedge designs (SWDs) were being used in the investigation of interventions to reduce opioid-related deaths in communities across the United States. However, these interventions are competing with external factors such as newly initiated public policies limiting opioid prescriptions, media awareness campaigns, and the COVID-19 pandemic. Furthermore, control communities may prematurely adopt components of the intervention as they become available. The presence of time-varying external factors that impact study outcomes is a well-known limitation of SWDs; common approaches to adjusting for them make use of a mixed effects modeling framework. However, these models have several shortcomings when external factors differentially impact intervention and control clusters.MethodsWe discuss limitations of commonly used mixed effects models in the context of proposed SWDs to investigate interventions intended to reduce opioid-related mortality, and propose extensions of these models to address these limitations. We conduct an extensive simulation study of anticipated data from SWD trials targeting the current opioid epidemic in order to examine the performance of these models in the presence of external factors. We consider confounding by time, premature adoption of intervention components, and time-varying effect modification— in which external factors differentially impact intervention and control clusters.ResultsIn the presence of confounding by time, commonly used mixed effects models yield unbiased intervention effect estimates, but can have inflated Type 1 error and result in under coverage of confidence intervals. These models yield biased intervention effect estimates when premature intervention adoption or effect modification are present. In such scenarios, models incorporating fixed intervention-by-time interactions with an unstructured covariance for intervention-by-cluster-by-time random effects result in unbiased intervention effect estimates, reach nominal confidence interval coverage, and preserve Type 1 error.ConclusionsMixed effects models can adjust for different combinations of external factors through correct specification of fixed and random time effects. Since model choice has considerable impact on validity of results and study power, careful consideration must be given to how these external factors impact study endpoints and what estimands are most appropriate in the presence of such factors.

Unbiased Estimates and Confidence Intervals for Riverine Loads

AbstractEstimating the uncertainty in annual riverine constituent loads, which is the mass that passes through a river cross‐section into a receiving water body, using infrequent water quality (WQ) observations is a difficult and unsolved task. Therefore, we propose an unbiased point estimation method and interval estimation method for river loads based on the rating curve (RC) method using importance sampling and the bootstrap method, respectively. In this paper, we first statistically explain the unbiasedness of load estimates using the proposed method. Second, the effectiveness of point and interval estimates by the proposed method is demonstrated for river loads from a small catchment and from large watersheds based on discharge and WQ data of solutes, nutrients, and suspended sediments with 10‐min to daily intervals. The results show that the proposed method provides unbiased estimates and appropriate coverage of confidence intervals regardless of the RC model used. The results also reveal that the dominant cause of bias in load estimates based on ordinary RC methods, such as the Loadest model, is not due to the poor simulation of observed loading rates by the RCs or because of the nonnormality of the regression residuals but rather improper sampling strategies. The proposed method is currently not feasible for WQ monitoring sites in large rivers due to the unmanageability of missing observations or censored data and an inefficient sampling strategy, although the requirement of unbiased estimation explained here can aid in scheduling high‐flow sampling for monitoring sites.

Machine Learning for Causal Inference: On the Use of Cross-fit Estimators.

Modern causal inference methods allow machine learning to be used to weaken parametric modeling assumptions. However, the use of machine learning may result in complications for inference. Doubly robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several different estimators for the average causal effect. The data generating mechanisms for the simulated treatment and outcome included log-transforms, polynomial terms, and discontinuities. We compared singly robust estimators (g-computation, inverse probability weighting) and doubly robust estimators (augmented inverse probability weighting, targeted maximum likelihood estimation). We estimated nuisance functions with parametric models and ensemble machine learning separately. We further assessed doubly robust cross-fit estimators. With correctly specified parametric models, all of the estimators were unbiased and confidence intervals achieved nominal coverage. When used with machine learning, the doubly robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage. Due to the difficulty of properly specifying parametric models in high-dimensional data, doubly robust estimators with ensemble learning and cross-fitting may be the preferred approach for estimation of the average causal effect in most epidemiologic studies. However, these approaches may require larger sample sizes to avoid finite-sample issues.

Deep Learning-based Propensity Scores for Confounding Control in Comparative Effectiveness Research: A Large-scale, Real-world Data Study.

Due to the non-randomized nature of real-world data, prognostic factors need to be balanced, which is often done by propensity scores (PSs). This study aimed to investigate whether autoencoders, which are unsupervised deep learning architectures, might be leveraged to compute PS. We selected patient-level data of 128,368 first-line treated cancer patients from the Flatiron Health EHR-derived de-identified database. We trained an autoencoder architecture to learn a lower-dimensional patient representation, which we used to compute PS. To compare the performance of an autoencoder-based PS with established methods, we performed a simulation study. We assessed the balancing and adjustment performance using standardized mean differences, root mean square errors (RMSE), percent bias, and confidence interval coverage. To illustrate the application of the autoencoder-based PS, we emulated the PRONOUNCE trial by applying the trial's protocol elements within an observational database setting, comparing two chemotherapy regimens. All methods but the manual variable selection approach led to well-balanced cohorts with average standardized mean differences <0.1. LASSO yielded on average the lowest deviation of resulting estimates (RMSE 0.0205) followed by the autoencoder approach (RMSE 0.0248). Altering the hyperparameter setup in sensitivity analysis, the autoencoder approach led to similar results as LASSO (RMSE 0.0203 and 0.0205, respectively). In the case study, all methods provided a similar conclusion with point estimates clustered around the null (e.g., HRautoencoder 1.01 [95% confidence interval = 0.80, 1.27] vs. HRPRONOUNCE 1.07 [0.83, 1.36]). Autoencoder-based PS computation was a feasible approach to control for confounding but did not perform better than some established approaches like LASSO.

One Instrument to Rule Them All: The Bias and Coverage of Just-Id IV

Two-stage least squares estimates in heavily over-identified instrumental variables (IV) models can be misleadingly close to the corresponding ordinary least squares (OLS) estimates when many instruments are weak. Just-identified (just-ID) IV estimates using a single instrument are also biased, but the importance of weak-instrument bias in just-ID IV applications remains contentious. We argue that in microeconometric applications, just-ID IV estimators can typically be treated as all but unbiased and that the usual inference strategies are likely to be adequate. The argument begins with contour plots for confidence interval coverage as a function of instrument strength and explanatory variable endogeneity. These show undercoverage in excess of 5% only for endogeneity beyond that seen even when IV and OLS estimates differ by an order of magnitude. Three widely cited microeconometric applications are used to explain why endogeneity is likely low enough for IV estimates to be reliable. We then show that an estimator that’s unbiased given a population first-stage sign restriction has bias exceeding that of IV when the restriction is imposed on the data. But screening on the sign of the estimated first stage is shown to halve the median bias of conventional IV without reducing coverage. To the extent that sign-screening is already part of empirical workflows, reported IV estimates enjoy the minimal bias of sign-screened just-ID IV.

COMPARISON OF CONVENTIONAL, BALANCED AND SUFFICIENT BOOTSTRAPPING APPROACHES VIA CONFIDENCE INTERVALS AND EFFICIENCY

There are various bootstrapping approaches depending on how bootstrap samples are selected. The conventional bootstrapping obtains random bootstrap samples by using all the units in the original sample. Balanced bootstrapping based on having individual observations with equal overall frequencies in all bootstrap samples and sufficient bootstrapping based on using only the distinct individual observations instead of all the units in the original sample are the two basic attempts proposed in this manner. This study compares the balanced, sufficient and conventional bootstrapping approaches in terms of efficiency, bootstrap confidence interval coverage accuracy, and average interval length. Although sufficient bootstrapping approach resulted in more efficient estimators and the narrower confidence intervals than the other two in all cases, none of the actual coverage level of confidence intervals was controlled within the desired limits. Conventional and balanced bootstrapping approaches have given quite similar results in terms of efficiency, coverage accuracy and average length.

Latent Logistic Interaction Modeling: A Simulation and Empirical Illustration of Type D Personality

ABSTRACT This study focuses on three popular methods to model interactions between two constructs containing measurement error in predicting an observed binary outcome: logistic regression using (1) observed scores, (2) factor scores, and (3) Structural Equation Modeling (SEM). It is still unclear how they compare with respect to bias and precision in the estimated interaction when item scores underlying the interaction constructs are skewed and ordinal. In this article, we investigated this issue using both a Monte Carlo simulation and an empirical illustration of the effect of Type D personality on cardiac events. Our results indicated that the logistic regression using SEM performed best in terms of bias and confidence interval coverage, especially at sample sizes of 500 or larger. Although for most methods bias increased when item scores were skewed and ordinal, SEM produced relatively unbiased interaction effect estimates when items were modeled as ordered categorical.

A Note on the Asymptotic Normality Theory of the Least Squares Estimates in Multivariate HAR-RV Models

In this work, multivariate heterogeneous autoregressive-realized volatility (HAR-RV) models are discussed with their least squares estimations. We consider multivariate HAR models of order p with q multiple assets to explore the relationships between two or more assets’ volatility. The strictly stationary solution of the HAR(p,q) model is investigated as well as the asymptotic normality theories of the least squares estimates are established in the cases of i.i.d. and correlated errors. In addition, an exponentially weighted multivariate HAR model with a common decay rate on the coefficients is discussed together with the common rate estimation. A Monte Carlo simulation is conducted to validate the estimations: sample mean and standard error of the estimates as well as empirical coverage and average length of confidence intervals are calculated. Lastly, real data of volatility of Gold spot price and S&amp;P index are applied to the model and it is shown that the bivariate HAR model fitted by selected optimal lags and estimated coefficients is well matched with the volatility of the financial data.

Approaches to addressing missing values, measurement error, and confounding in epidemiologic studies

ObjectivesEpidemiologic studies often suffer from incomplete data, measurement error (or misclassification), and confounding. Each of these can cause bias and imprecision in estimates of exposure–outcome relations. We describe and compare statistical approaches that aim to control all three sources of bias simultaneously. Study Design and SettingWe illustrate four statistical approaches that address all three sources of bias, namely, multiple imputation for missing data and measurement error, multiple imputation combined with regression calibration, full information maximum likelihood within a structural equation modeling framework, and a Bayesian model. In a simulation study, we assess the performance of the four approaches compared with more commonly used approaches that do not account for measurement error, missing values, or confounding. ResultsThe results demonstrate that the four approaches consistently outperform the alternative approaches on all performance metrics (bias, mean squared error, and confidence interval coverage). Even in simulated data of 100 subjects, these approaches perform well. ConclusionThere can be a large benefit of addressing measurement error, missing values, and confounding to improve the estimation of exposure–outcome relations, even when the available sample size is relatively small.

A Structured Approach to Evaluating Life-Course Hypotheses: Moving Beyond Analyses of Exposed Versus Unexposed in the -Omics Context.

The structured life-course modeling approach (SLCMA) is a theory-driven analytical method that empirically compares multiple prespecified life-course hypotheses characterizing time-dependent exposure-outcome relationships to determine which theory best fits the observed data. In this study, we performed simulations and empirical analyses to evaluate the performance of the SLCMA when applied to genomewide DNA methylation (DNAm). Using simulations (n=700), we compared 5 statistical inference tests used with SLCMA, assessing the familywise error rate, statistical power, and confidence interval coverage to determine whether inference based on these tests was valid in the presence of substantial multiple testing and small effects-2 hallmark challenges of inference from -omics data. In the empirical analyses (n=703), we evaluated the time-dependent relationship between childhood abuse and genomewide DNAm. In simulations, selective inference and the max-|t|-test performed best: Both controlled the familywise error rate and yielded moderate statistical power. Empirical analyses using SLCMA revealed time-dependent effects of childhood abuse on DNAm. Our findings show that SLCMA, applied and interpreted appropriately, can be used in high-throughput settings to examine time-dependent effects underlying exposure-outcome relationships over the life course. We provide recommendations for applying the SLCMA in -omics settings and encourage researchers to move beyond analyses of exposed versus unexposed individuals.

Erratum to: Large-Scale Evidence Generation and Evaluation across a Network of Databases (LEGEND): Assessing Validity Using Hypertension as a Case Study.

OBJECTIVES To demonstrate the application of the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) principles described in our companion article to hypertension treatments and assess internal and external validity of the generated evidence. MATERIALS AND METHODS LEGEND defines a process for high-quality observational research based on 10 guiding principles. We demonstrate how this process, here implemented through large-scale propensity score modeling, negative and positive control questions, empirical calibration, and full transparency, can be applied to compare antihypertensive drug therapies. We assess internal validity through covariate balance, confidence-interval coverage, between-database heterogeneity, and transitivity of results. We assess external validity through comparison to direct meta-analyses of randomized controlled trials (RCTs). RESULTS From 21.6 million unique antihypertensive new users, we generate 6 076 775 effect size estimates for 699 872 research questions on 12 946 treatment comparisons. Through propensity score matching, we achieve balance on all baseline patient characteristics for 75% of estimates, observe 95.7% coverage in our effect-estimate 95% confidence intervals, find high between-database consistency, and achieve transitivity in 84.8% of triplet hypotheses. Compared with meta-analyses of RCTs, our results are consistent with 28 of 30 comparisons while providing narrower confidence intervals. CONCLUSION We find that these LEGEND results show high internal validity and are congruent with meta-analyses of RCTs. For these reasons we believe that evidence generated by LEGEND is of high quality and can inform medical decision-making where evidence is currently lacking. Subsequent publications will explore the clinical interpretations of this evidence.

Comparing cross-classified mixed effects and Bayesian structural equations modeling for stimulus sampling designs: A simulation study

Researchers examining a wide range of psychological phenomena, including interpersonal perception, attitude formation, and stereotype activation, apply stimulus sampling designs (SSD). The standard SSD study requires participant raters to provide evaluations of a series of target stimuli (e.g., photographs, media clips, vignettes), and the constituent responses are simultaneously nested within participants and stimuli, yielding a cross-classified data structure. Prior methodological work has illustrated the application of both cross-classified mixed effects, and cross-classified structural equation modeling to accommodate the corresponding dependency structure. Despite their widespread application, little is known about how sample size for both participants and stimuli is associated with inferential power and coverage in SSDs. Even less is known about the feasibility of random slopes, or whether the frequentist (maximum likelihood) and Bayesian (MCMC) estimators differ in accuracy or efficiency under the design conditions typically observed in SSD studies. We conducted a Monte Carlo simulation study to better understand parameter bias, statistical power, and confidence or credible interval coverage, as a function of the number of participant raters and target stimuli, effect size, as well as the presence of random slopes, and modeling framework. Recommendations for future research are provided.

Performance of unanchored matching-adjusted indirect comparison (MAIC) for the evidence synthesis of single-arm trials with time-to-event outcomes

BackgroundThe objectives of the present study were to evaluate the performance of a time-to-event data reconstruction method, to assess the bias and efficiency of unanchored matching-adjusted indirect comparison (MAIC) methods for the analysis of time-to-event outcomes, and to propose an approach to adjust the bias of unanchored MAIC when omitted confounders across trials may exist.MethodsTo evaluate the methods using a Monte Carlo approach, a thousand repetitions of simulated data sets were generated for two single-arm trials. In each repetition, researchers were assumed to have access to individual-level patient data (IPD) for one of the trials and the published Kaplan-Meier curve of another. First, we compared the raw data and the reconstructed IPD using Cox regressions to determine the performance of the data reconstruction method. Then, we evaluated alternative unanchored MAIC strategies with varying completeness of covariates for matching in terms of bias, efficiency, and confidence interval coverage. Finally, we proposed a bias factor-adjusted approach to gauge the true effects when unanchored MAIC estimates might be biased due to omitted variables.ResultsReconstructed data sufficiently represented raw data in the sense that the difference between the raw and reconstructed data was not statistically significant over the one thousand repetitions. Also, the bias of unanchored MAIC estimates ranged from minimal to substantial as the set of covariates became less complete. More, the confidence interval estimates of unanchored MAIC were suboptimal even using the complete set of covariates. Finally, the bias factor-adjusted method we proposed substantially reduced omitted variable bias.ConclusionsUnanchored MAIC should be used to analyze time-to-event outcomes with caution. The bias factor may be used to gauge the true treatment effect.