Unmeasured Covariates Research Articles

Correlates of improved outcomes in patients with COVID-19 treated in US emergency departments

BackgroundThe COVID-19 pandemic was managed in part by the rapid development of vaccines, diagnostics, and therapeutics including antiviral agents and advances in emergency airway and ventilatory management. The impact of these therapeutic advances on clinically pertinent metrics of emergency care have not been well-studied. MethodsWe abstracted data from emergency department (ED) visits made to 21 US health systems during the first two years of the pandemic, from February 1, 2020 to January 31, 2022. These health systems were participants in the NIH-supported COVID EHR Cohort, in which the University of Wisconsin served as the coordinating site. Limited patient-level data files were submitted monthly. Data elements included demographic and clinical variables, as well as standard measures of ED outcomes including 72-h returns, 72-h returns leading to readmission, and in-hospital mortality. Multivariable models were fitted to identify correlates of each of the dependent variables. A test for trend was used to detect changes in outcomes over time. ResultsDuring the two-year period, 150,357 individuals aged 18 years or older visited the ED. The median age was 45.4 years (IQR 27), 58.1 % were female, 49 % were White, 18.3 % Hispanic/Latino, and 45 % were publicly insured or uninsured. The prevalence of 72-h ED returns, readmissions, and in-hospital mortality significantly declined across the two-year period. SARS-CoV-2 vaccination was associated with reduced ED returns and mortality. Therapeutic agents were associated with increased mortality risk but were likely confounded by unmeasured covariates. ConclusionsOperational and clinical outcomes of ED-based treatment of individuals with COVID-19 improved in the first two years of the pandemic. This improvement is likely multifactorial and includes the development and deployment of SARS-CoV-2-specific vaccines, therapeutic agents, and improved healthcare delivery in the ED and elsewhere addressing management of airway and ventilatory status, as well as increased innate immunity in the general population.

Two assumptions of the prior event rate ratio approach for controlling confounding can be evaluated by self-controlled case series and dynamic random intercept modeling

ObjectivesThe prior event rate ratio (PERR) is a recently developed approach for controlling confounding by measured and unmeasured covariates in real-world evidence research and observational studies. Despite its rising popularity in studies of safety and effectiveness of biopharmaceutical products, there is no guidance on how to empirically evaluate its model assumptions. We propose two methods to evaluate two of the assumptions required by the PERR, specifically, the assumptions that occurrence of outcome events does not alter the likelihood of receiving treatment, and that earlier event rate does not affect later event rate. Study Design and SettingWe propose using self-controlled case series (SCCS) and dynamic random intercept modeling (DRIM), respectively, to evaluate the two aforementioned assumptions. A nonmathematical introduction of the methods and their application to evaluate the assumptions are provided. We illustrate the evaluation with secondary analysis of deidentified data on pneumococcal vaccination and clinical pneumonia in The Gambia, West Africa. ResultsSCCS analysis of data on 12,901 vaccinated Gambian infants did not reject the assumption of clinical pneumonia episodes had no influence on the likelihood of pneumococcal vaccination. DRIM analysis of 14,325 infants with a total of 1719 episodes of clinical pneumonia did not reject the assumption of earlier episodes of clinical pneumonia had no influence on later incidence of the disease. ConclusionThe SCCS and DRIM methods can facilitate appropriate use of the PERR approach to control confounding. Plain Language SummaryThe prior event rate ratio is a promising approach for analysis of real-world data and observational studies. We propose two statistical methods to evaluate the validity of two assumptions it is based on. They can facilitate appropriate use of the prior even rate ratio.

Robustness of effect modification estimates from two-stage multi-location designs to unmeasured covariates: A simulation study

Robustness of effect modification estimates from two-stage multi-location designs to unmeasured covariates: A simulation study

Valid instrumental variable selection method using negative control outcomes and constructing efficient estimator.

In observational studies, instrumental variable (IV) methods are commonly applied when there are unmeasured covariates. In Mendelian randomization, constructing an allele score using many single nucleotide polymorphisms is often implemented; however, estimating biased causal effects by including some invalid IVs poses some risks. Invalid IVs are those IV candidates that are associated with unobserved variables. To solve this problem, we developed a novel strategy using negative control outcomes (NCOs) as auxiliary variables. Using NCOs, we are able to select only valid IVs and exclude invalid IVs without knowing which of the instruments are invalid. We also developed a new two-step estimation procedure and proved the semiparametric efficiency of our estimator. The performance of our proposed method was superior to some previous methods through simulations. Subsequently, we applied the proposed method to the UK Biobank dataset. Our results demonstrate that the use of an auxiliary variable, such as an NCO, enables the selection of valid IVs with assumptions different from those used in previousmethods.

Bayesian Shared Component Spatial Modeling for Assessing the Shared and Age Group-Specific Mental Health Disorder Risk of Young and Old Age Groups: A Case Study of Toronto Neighborhoods, Canada

Mental health disorder risks of young and old age groups hold considerable importance for understanding present and future risk burdens. However, assessing mental health risks is significantly constrained by the influence of shared and age group-specific spatial processes and risk factors. Therefore, this study employed Bayesian shared component spatial modeling (BSCSM) to analyze mental health disorder data obtained from young (20–44 years) and old (65+ years) age groups in Toronto. BSCSM was employed to model the shared and age group-specific disorder risk and to identify hotspot areas. The unmeasured covariates, overdispersion, and latent spatial processes were adjusted using spatial and non-spatial random effect terms. The findings from BSCSM were finally compared with non-shared component modeling approaches. The results suggest that over 60% of variations in mental health disorder risk for both age groups could be explained by the shared component. The high-risk neighborhoods were mainly localized in southern and north-central Toronto for the young and old age groups. Deviance information criterion values suggested that models from BSCSM outperformed non-BSCSM models. BSCSM risk maps were also better at identifying high-risk areas. This work demonstrated that both shared and age group-specific risks are essential for assessing mental health disorder risk and devising targeted interventions.

Semi-supervised mixture multi-source exchangeability model for leveraging real-world data in clinical trials.

The traditional trial paradigm is often criticized as being slow, inefficient, and costly. Statistical approaches that leverage external trial data have emerged to make trials more efficient by augmenting the sample size. However, these approaches assume that external data are from previously conducted trials, leaving a rich source of untapped real-world data (RWD) that cannot yet be effectively leveraged. We propose a semi-supervised mixture (SS-MIX) multisource exchangeability model (MEM); a flexible, two-step Bayesian approach for incorporating RWD into randomized controlled trial analyses. The first step is a SS-MIX model on a modified propensity score and the second step is a MEM. The first step targets a representative subgroup of individuals from the trial population and the second step avoids borrowing when there are substantial differences in outcomes among the trial sample and the representative observational sample. When comparing the proposed approach to competing borrowing approaches in a simulation study, we find that our approach borrows efficiently when the trial and RWD are consistent, while mitigating bias when the trial and external data differ on either measured or unmeasured covariates. We illustrate the proposed approach with an application to a randomized controlled trial investigating intravenous hyperimmune immunoglobulin in hospitalized patients with influenza, while leveraging data from an external observational study to supplement a subgroup analysis by influenza subtype.

Can We Reliably Detect Biases that Matter in Observational Studies?

In an observational study of the effects caused by a treatment, biases from unmeasured covariates remain a concern even after successful adjustments for measured covariates. This concern is partly addressed by demonstrating that the qualitative conclusions of the primary analysis would not be altered by small or moderate biases—that these conclusions are insensitive to small or moderate bias. Additionally, the concern is partly addressed by collecting additional information, such as outcomes known to be unaffected by the treatment, and using this information as a test of various biases. Is there a gap between these two activities? Perhaps the study is insensitive to small biases, and we can detect large biases, but the study is sensitive to moderate biases that cannot be detected—that is an informal description of a gap. The concept of “no gap” is defined formally in Definition 3.1, and the probability of “no gap” is determined under various sampling situations. When there is no gap, ask: Are causal conclusions measurably strengthened? If so, by how much? The answer depends upon the covering design sensitivity, Γ⌢, defined to be the smallest bias that can explain both the ostensible effect of the treatment on the primary outcome and the evidence of bias provided by the unaffected outcome. The covering design sensitivity is calculated in various contexts. A small observational study of the effects of light alcohol consumption on HDL cholesterol is used to illustrate ideas and methods.

A Note on Stratification Errors in the Analysis of Clinical Trials

Stratification in both the design and analysis of randomized clinical trials is common. Despite features in automated randomization systems to reconfirm the stratifying variables, incorrect values of these variables may be entered. These errors are often detected during subsequent data collection and verification. Questions remain about whether to use the mis-reported initial stratification or the corrected values in subsequent analyses. It is shown that the likelihood function resulting from the design of randomized clinical trials supports the use of the corrected values. New definitions are proposed that characterize misclassification errors as “ignorable” and “nonignorable.” Ignorable errors may depend on the correct strata and any other modeled baseline covariates, but they are otherwise unrelated to potential treatment outcomes. Data management review suggests most misclassification errors are arbitrarily produced by distracted investigators, so they are ignorable or at most weakly dependent on measured and unmeasured baseline covariates. Ignorable misclassification errors may produce a small increase in standard errors, but other properties of the planned analyses are unchanged (e.g., unbiasedness, confidence interval coverage). It is shown that unbiased linear estimation in the absence of misclassification errors remains unbiased when there are non-ignorable misclassification errors, and the corresponding confidence intervals based on the corrected strata values are conservative.

Impact of a Comprehensive ST‐Segment–Elevation Myocardial Infarction Protocol on Key Process Metrics in Black Americans

Impact of a Comprehensive ST‐Segment–Elevation Myocardial Infarction Protocol on Key Process Metrics in Black Americans

Spatially varying coefficients can improve parsimony and descriptive power for species distribution models

Species distribution models (SDMs) are widely used to relate species occurrence and density to local environmental conditions, and often include a spatially correlated variable to account for spatial patterns in residuals. Ecologists have extended SDMs to include spatially varying coefficients (SVCs), where the response to a given covariate varies smoothly over space and time. However, SVCs see relatively little use perhaps because they remain less known relative to other SDM techniques. We therefore review ecological contexts where SVCs can improve the interpretability and descriptive power from SDMs, including local responses to regional indices that represent ecological teleconnections; density‐dependent habitat selection; spatially varying detectability; and context‐dependent covariate responses that represent interactions with unmeasured covariates. We then illustrate three additional examples in detail using the vector autoregressive spatio‐temporal (VAST) model. First, a spatially varying decadal trends model identifies decadal trends for arrowtooth flounder Atheresthes stomias density in the Bering Sea from 1982 to 2019. Second, a trait‐based joint SDM highlights the role of body size and temperature in spatial community assembly in the Gulf of Alaska. Third, an age‐structured SDM for walleye pollock Gadus chalcogrammus in the Bering Sea contrasts cohorts with broad spatial distributions (1996 and 2009) and those that are more spatially constrained (2002 and 2015). We conclude that SVCs extend SDMs to address a wide variety of ecological contexts and can be used to better understand a range of ecological processes, e.g. density dependence, community assembly and population dynamics.

Estimating and improving dynamic treatment regimes with a time-varying instrumental variable

Abstract Estimating dynamic treatment regimes (DTRs) from retrospective observational data is challenging as some degree of unmeasured confounding is often expected. In this work, we develop a framework of estimating properly defined ‘optimal’ DTRs with a time-varying instrumental variable (IV) when unmeasured covariates confound the treatment and outcome, rendering the potential outcome distributions only partially identified. We derive a novel Bellman equation under partial identification, use it to define a generic class of estimands (termed IV-optimal DTRs) and study the associated estimation problem. We then extend the IV-optimality framework to tackle the policy improvement problem, delivering IV-improved DTRs that are guaranteed to perform no worse and potentially better than a prespecified baseline DTR. Importantly, this IV-improvement framework opens up the possibility of strictly improving upon DTRs that are optimal under the no unmeasured confounding assumption (NUCA). We demonstrate via extensive simulations the superior performance of IV-optimal and IV-improved DTRs over the DTRs that are optimal only under the NUCA. In a real data example, we embed retrospective observational registry data into a natural, two-stage experiment with noncompliance using a differential-distance-based, time-varying IV and estimate useful IV-optimal DTRs that assign mothers to a high-level or low-level neonatal intensive care unit based on their prognostic variables.

Relationship between hysterectomy and stroke in National Health and Nutrition Examination Survey (NHANES) 2007–2018: a cross-sectional study

Hysterectomy is the most common type of gynecological operation in the United States. However, complications can occur during or after the operation. Some studies suggest that hysterectomy may increase the risk of stroke. However, other studies have found different conclusions on this matter. This inconsistent conclusion may be due to small sample sizes or limited covariates. So, we sought to further investigate the correlation between hysterectomy and stroke. Our analysis was based on the data from 2007-2018 National Health and Nutrition Examination Survey (NHANES). We excluded participants with missing hysterectomy data (Question "Have you had a hysterectomy, including a partial hysterectomy, that is, surgery to remove {your/her} uterus or womb?"), participants with missing stroke data (Question "Has a doctor or other health professional ever told you that you had a stroke?), a total of 15,241 participants were included in our analysis. To estimate the correlation between hysterectomy and stroke, logistic regression models were used after adjusting for sociodemographic and health-related factors, including age, race, education level, marital status, annual family income, body mass index (BMI), alcohol consumption in the past 12 months, having smoked at least 100 cigarettes in a lifetime, hypertension, hypercholesterolemia, and diabetes. The unadjusted model suggests that women who had undergone a hysterectomy were 3.15 [95% confidence interval (CI): 2.67-3.71] times more likely to have a stroke than women who had not undergone a hysterectomy. In the crude and fully-adjusted models, the correlation between hysterectomy and stroke was consistent [odds ratio (OR)crude-adjusted =1.55 (95% CI: 1.30-1.85), ORfully-adjusted =1.36 (95% CI: 1.14-1.63)]. In the subgroup analysis stratified by age, hysterectomy seemed to have more risk for stroke occurrence regardless of subgroup, even after adjusting sociodemographic and health-related factors. Interestingly, the women who were less than or equal to 50 years old had greater odds of stroke (ORfully-adjusted =1.96) compared with women who were aged older than 50 (ORfully-adjusted =1.42). In our study, we concluded hysterectomy may increase the risk of stroke. However, as our study is a cross-sectional study and unmeasured covariates may still exist, more researches are required to confirm this conclusion.

Propensity scores in the design of observational studies for causal effects

Summary The design of any study, whether experimental or observational, that is intended to estimate the causal effects of a treatment condition relative to a control condition refers to those activities that precede any examination of outcome variables. As defined in our 1983 article (Rosenbaum & Rubin, 1983), the propensity score is the unit-level conditional probability of assignment to treatment versus control given the observed covariates; so the propensity score explicitly does not involve any outcome variables, in contrast to other summaries of variables sometimes used in observational studies. Balancing the distributions of covariates in the treatment and control groups by matching or balancing on the propensity score is therefore an aspect of the design of the observational study. In this invited comment on our 1983 article, we review the situation in the early 1980s and recall some apparent paradoxes that propensity scores helped to resolve. We demonstrate that it is possible to balance an enormous number of low-dimensional summaries of a high-dimensional covariate, even though it is generally impossible to match individuals closely for all the components of a high-dimensional covariate. In a sense, there is only one crucial observed covariate, the propensity score, and there is one crucial unobserved covariate, the principal unobserved covariate. The propensity score and the principal unobserved covariate are equal when treatment assignment is strongly ignorable, that is, unconfounded. Controlling for observed covariates is a prelude to the crucial step from association to causation, the step that addresses potential biases from unmeasured covariates. The design of an observational study also prepares for the step to causation: by selecting comparisons to increase the design sensitivity, by seeking opportunities to detect bias, by seeking mutually supportive evidence affected by different biases, by incorporating quasi-experimental devices such as multiple control groups, and by including the economist’s instruments. All of these considerations reflect the formal development of sensitivity analyses that were largely informal prior to the 1980s.

The challenge of pre-enrollment selection bias in estimating the effects of air pollution on cognitive decline and other dementia-related outcomes

BACKGROUND AND AIM: Research on air pollution’s (AP) association with dementia-related outcomes has proliferated; however, selection bias is a concern. Past research has addressed selection bias from differential post-enrollment attrition, but differential pre-enrollment selection may also be influential depending on who survives to the eligible age of enrollment and enrolls. In this project, we quantify potential biases arising from differential selection into studies of older adults through simulations based on real-world studies of AP and cognitive decline. METHODS: We simulated 1,000 cohorts of 100,000 hypothetical participants. Participants were assigned a realistic AP exposure based on prior studies, plus an unmeasured binary covariate value (U). Our simulation models specified that AP had no effect on cognitive decline, whereas U had a small positive effect. We selected 10,000 participants from each cohort to form 1,000 analytic samples, where the probability of selection depended on both AP and U. We quantified the magnitude of the resulting "collider-stratification” selection bias by comparing the average estimated effect of AP on cognitive decline across our analytic samples with the true (unbiased) value of zero. We also explored whether these biases were sensitive to AP-U interactions. RESULTS: The association between AP and cognitive decline was upwardly biased when lower AP exposure and the presence of U increased the probability of selection. The bias magnitude was particularly sensitive to multiplicative AP-U interactions. Importantly, we could replicate prior observational study findings without resorting to extreme parameter values. CONCLUSIONS: Pre-enrollment selection bias can lead to qualitatively inaccurate effect estimates in studies of older adults. These cohorts are often highly selected by virtue of their survival to eligible study age and willingness to enroll. Bias analyses (e.g., our simulation-based approach) can help identify key assumption violations that could nullify or reverse environmental effect estimates. KEYWORDS: Air pollution; Cognitive decline; Selection bias

Periconception Red Blood Cell Folate and Offspring Congenital Heart Disease : Nested Case-Control and Mendelian Randomization Studies.

Periconception folic acid supplementation has been suggested to protect against congenital heart disease (CHD), but the association between maternal red blood cell (RBC) folate, the gold-standard biomarker of folate exposure, and subsequent offspring CHD risk is lacking. To quantify the association between periconception maternal RBC folate and offspring CHD risk. Prospective, nested, case-control study and 1-sample Mendelian randomization. (ClinicalTrials.gov: NCT02737644). 29 maternity institutions in 12 districts of Greater Shanghai, China. All 197 mothers of offspring with CHD and 788 individually matched mothers of unaffected offspring from the SPCC (Shanghai Preconception Cohort). Maternal RBC folate was measured before or at early pregnancy. Odds ratios [ORs] were estimated using conditional logistic regression after adjustment for covariates. Mendelian randomization was done using the methylenetetrahydrofolate reductase (MTHFR) C677T as the genetic instrument. Case patients had lower median maternal RBC folate concentrations than control participants (714 nmol/L [interquartile range, 482 to 1008 nmol/L] vs. 788 nmol/L [557 to 1094 nmol/L]). Maternal RBC folate concentrations were inversely associated with offspring CHD (adjusted OR per 100 nmol/L, 0.93 [95% CI, 0.89 to 0.99]). The adjusted OR for mothers with periconception RBC folate of 906 nmol/L or more (vs. <906 nmol/L) was 0.61 (CI, 0.40 to 0.93). Mendelian randomization showed that each 100-nmol increase in maternal RBC folate concentrations was significantly associated with reduced offspring CHD risk (OR, 0.75 [CI, 0.61 to 0.92]). Potential confounding due to unmeasured covariates in the nested case-control study. Higher maternal RBC folate is associated with reduced offspring CHD risk. For primary CHD prevention, higher target RBC folate levels than currently recommended for neural tube defect prevention may be needed and warrant further study. National Key Research and Development Program of China, National Natural Science Foundation of China, China Postdoctoral Science Foundation, and Chinese Academy of Medical Sciences Innovation Fund for Medical Sciences.

Key considerations for choosing a statistical method to deal with incomplete treatment adherence in pragmatic trials.

Pragmatic trials offer practical means of obtaining real-world evidence to help improve decision-making in comparative effectiveness settings. Unfortunately, incomplete adherence is a common problem in pragmatic trials. The commonly used methods in randomized control trials often cannot handle the added complexity imposed by incomplete adherence, resulting in biased estimates. Several naive methods and advanced causal inference methods (e.g., inverse probability weighting and instrumental variable-based approaches) have been used in the literature to deal with incomplete adherence. Practitioners and applied researchers are often confused about which method to consider under a given setting. This current work is aimed to review commonly used statistical methods to deal with non-adherence along with their key assumptions, advantages, and limitations, with a particular focus on pragmatic trials. We have listed the applicable settings for these methods and provided a summary of available software. All methods were applied to two hypothetical datasets to demonstrate how these methods perform in a given scenario, along with the R codes. The key considerations include the type of intervention strategy (point treatment settings, where treatment is administered only once versus sustained treatment settings, where treatment has to be continued over time) and availability of data (e.g., the extent of measured or unmeasured covariates that are associated with adherence, dependent confounding impacted by past treatment, and potential violation of assumptions). This study will guide practitioners and applied researchers to use the appropriate statistical method to address incomplete adherence in pragmatic trial settings for both the point and sustained treatment strategies.

A Review of Causal Inference for External Comparator Arm Studies.

Randomized controlled trials (RCTs) are the gold standard design to establish the efficacy of new drugs and to support regulatory decision making. However, a marked increase in the submission of single-arm trials (SATs) has been observed in recent years, especially in the field of oncology due to the trend towards precision medicine contributing to the rise of new therapeutic interventions for rare diseases. SATs lack results for control patients, and information from external sources can be compiled to provide context for better interpretability of study results. External comparator arm (ECA) studies are defined as a clinical trial (most commonly a SAT) and an ECA of a comparable cohort of patients-commonly derived from real-world settings including registries, natural history studies, or medical records of routine care. This publication aims to provide a methodological overview, to sketch emergent best practice recommendations and to identify future methodological research topics. Specifically, existing scientific and regulatory guidance for ECA studies is reviewed and appropriate causal inference methods are discussed. Further topics include sample size considerations, use of estimands, handling of different data sources regarding differential baseline covariate definitions, differential endpoint measurements and timings. In addition, unique features of ECA studies are highlighted, specifically the opportunity to address bias caused by unmeasured ECA covariates, which are available in the SAT.

Bayesian adaptive randomization design incorporating propensity score-matched historical controls.

Incorporating historical control data to augment the control arm in randomized controlled trials (RCTs) is one way of increasing their efficiency and feasibility when adequate RCTs cannot be conducted. In recent work, a Bayesian adaptive randomization design incorporating historical control data has been proposed to reduce sample size according to the amount of information that could be borrowed, assessed at interim assessment in respect to prior-data conflict. However, the approach does not distinguish between the two sources of prior-data conflict: (1) imbalance in measured covariates, and (2) imbalance in unmeasured covariates. In this paper, we propose an extension of the Bayesian adaptive randomization design to incorporate propensity score-matched historical controls. At interim assessment, historical controls similar to the concurrent controls in terms of measured covariates are selected using propensity score matching. Then, final sample size of the control arm is adjusted according to the extent of borrowing from the matched historical controls quantified by effective historical sample size. The conditional power prior approach and commensurate prior approach are adopted for designing the prior, and addressing prior-data conflict due to unmeasured covariate imbalance. Simulation results show that the proposed method yields reduced bias in treatment effect estimates, type I error at the nominal level, and reduced sample size while maintaining statistical power. Even when residual imbalance exists due to unmeasured covariates, the proposed method borrowed more information without risking substantially inflated type I error and bias, providing meaningful implications for use of historical controls to facilitate the conduct of adequate RCTs.

Improving the causal treatment effect estimation with propensity scores by the bootstrap

When observational studies are used to establish the causal effects of treatments, the estimated effect is affected by treatment selection bias. The inverse propensity score weight (IPSW) is often used to deal with such bias. However, IPSW requires strong assumptions whose misspecifications and strategies to correct the misspecifications were rarely studied. We present a bootstrap bias correction of IPSW (BC-IPSW) to improve the performance of propensity score in dealing with treatment selection bias in the presence of failure to the ignorability and overlap assumptions. The approach was motivated by a real observational study to explore the potential of anticoagulant treatment for reducing mortality in patients with end-stage renal disease. The benefit of the treatment to enhance survival was demonstrated; the suggested BC-IPSW method indicated a statistically significant reduction in mortality for patients receiving the treatment. Using extensive simulations, we show that BC-IPSW substantially reduced the bias due to the misspecification of the ignorability and overlap assumptions. Further, we showed that IPSW is still useful to account for the lack of treatment randomization, but its advantages are stringently linked to the satisfaction of ignorability, indicating that the existence of relevant though unmeasured or unused covariates can worsen the selection bias.

Partially Pooled Propensity Score Models for Average Treatment Effect Estimation with Multilevel Data

Abstract Causal inference analyses often use existing observational data, which in many cases has some clustering of individuals. In this paper, we discuss propensity score weighting methods in a multilevel setting where within clusters individuals share unmeasured confounders that are related to treatment assignment and the potential outcomes. We focus in particular on settings where models with fixed cluster effects are either not feasible or not useful due to the presence of a large number of small clusters. We found, both through numerical experiments and theoretical derivations, that a strategy of grouping clusters with similar treatment prevalence and estimating propensity scores within such cluster groups is effective in reducing bias from unmeasured cluster-level covariates under mild conditions on the outcome model. We apply our proposed method in evaluating the effectiveness of centre-based pre-school programme participation on children’s achievement at kindergarten, using the Early Childhood Longitudinal Study Kindergarten data.