Performance Of Propensity Score Methods Research Articles

RWD24 Assessing the Performance of Propensity Score Methods in Balancing Unobservable Confounders: Evidence From Linked Data in Multiple Sclerosis

We used a database in multiple sclerosis (MS) linking claims to electronic medical records (EMRs) to investigate whether propensity score (PS) methods implemented on claims alone are successful in balancing the expanded disability status scale (EDSS) measured in EMRs only.

Logistic regression frequently outperformed propensity score methods, especially for large datasets: a simulation study

ObjectivesIn observational studies, researchers must select a method to control for confounding. Options include propensity score (PS) methods and regression. It remains unclear how dataset characteristics (size, overlap in PSs, and exposure prevalence) influence the relative performance of the methods. Study Design and SettingA simulation study to evaluate the role of dataset characteristics on the performance of PS methods, compared to logistic regression, for estimating a marginal odds ratio was conducted. Dataset size, overlap in PSs, and exposure prevalence were varied. ResultsRegression showed poor coverage for small sample sizes, but with large sample sizes was relatively robust to imbalance in PSs and low exposure prevalence. PS methods displayed suboptimal coverage as overlap in PSs decreased, which was exacerbated at larger sample sizes. Power of matching methods was particularly affected by a lack of overlap, low exposure prevalence, and small sample size. The advantage of regression for large data size was reduced in sensitivity analysis with a complementary log–log outcome generation mechanism and unmeasured confounding, with superior bias and error but inferior coverage to matching methods. ConclusionDataset characteristics influence performance of methods for confounder adjustment. In many scenarios, regression may be the preferable option.

Unicompartmental compared with total knee replacement for patients with multimorbidities: a cohort study using propensity score stratification and inverse probability weighting.

Although routine NHS data potentially include all patients, confounding limits their use for causal inference. Methods to minimise confounding in observational studies of implantable devices are required to enable the evaluation of patients with severe systemic morbidity who are excluded from many randomised controlled trials. Stage 1 - replicate the Total or Partial Knee Arthroplasty Trial (TOPKAT), a surgical randomised controlled trial comparing unicompartmental knee replacement with total knee replacement using propensity score and instrumental variable methods. Stage 2 - compare the risk benefits and cost-effectiveness of unicompartmental knee replacement with total knee replacement surgery in patients with severe systemic morbidity who would have been ineligible for TOPKAT using the validated methods from stage 1. This was a cohort study. Data were obtained from the National Joint Registry database and linked to hospital inpatient (Hospital Episode Statistics) and patient-reported outcome data. Stage 1 - people undergoing unicompartmental knee replacement surgery or total knee replacement surgery who met the TOPKAT eligibility criteria. Stage 2 - participants with an American Society of Anesthesiologists grade of ≥ 3. The patients were exposed to either unicompartmental knee replacement surgery or total knee replacement surgery. The primary outcome measure was the postoperative Oxford Knee Score. The secondary outcome measures were 90-day postoperative complications (venous thromboembolism, myocardial infarction and prosthetic joint infection) and 5-year revision risk and mortality. The main outcome measures for the health economic analysis were health-related quality of life (EuroQol-5 Dimensions) and NHS hospital costs. In stage 1, propensity score stratification and inverse probability weighting replicated the results of TOPKAT. Propensity score adjustment, propensity score matching and instrumental variables did not. Stage 2 included 2256 unicompartmental knee replacement patients and 57,682 total knee replacement patients who had severe comorbidities, of whom 145 and 23,344 had linked Oxford Knee Scores, respectively. A statistically significant but clinically irrelevant difference favouring unicompartmental knee replacement was observed, with a mean postoperative Oxford Knee Score difference of < 2 points using propensity score stratification; no significant difference was observed using inverse probability weighting. Unicompartmental knee replacement more than halved the risk of venous thromboembolism [relative risk 0.33 (95% confidence interval 0.15 to 0.74) using propensity score stratification; relative risk 0.39 (95% confidence interval 0.16 to 0.96) using inverse probability weighting]. Unicompartmental knee replacement was not associated with myocardial infarction or prosthetic joint infection using either method. In the long term, unicompartmental knee replacement had double the revision risk of total knee replacement [hazard ratio 2.70 (95% confidence interval 2.15 to 3.38) using propensity score stratification; hazard ratio 2.60 (95% confidence interval 1.94 to 3.47) using inverse probability weighting], but half of the mortality [hazard ratio 0.52 (95% confidence interval 0.36 to 0.74) using propensity score stratification; insignificant effect using inverse probability weighting]. Unicompartmental knee replacement had lower costs and higher quality-adjusted life-year gains than total knee replacement for stage 2 participants. Although some propensity score methods successfully replicated TOPKAT, unresolved confounding may have affected stage 2. Missing Oxford Knee Scores may have led to information bias. Propensity score stratification and inverse probability weighting successfully replicated TOPKAT, implying that some (but not all) propensity score methods can be used to evaluate surgical innovations and implantable medical devices using routine NHS data. Unicompartmental knee replacement was safer and more cost-effective than total knee replacement for patients with severe comorbidity and should be considered the first option for suitable patients. Further research is required to understand the performance of propensity score methods for evaluating surgical innovations and implantable devices. This trial is registered as EUPAS17435. This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 66. See the NIHR Journals Library website for further project information.

Subgroup balancing propensity score.

This paper concerns estimation of subgroup treatment effects with observational data. Existing propensity score methods are mostly developed for estimating overall treatment effect. Although the true propensity scores balance covariates in any subpopulations, the estimated propensity scores may result in severe imbalance in subgroup samples. Indeed, subgroup analysis amplifies a bias-variance tradeoff, whereby increasing complexity of the propensity score model may help to achieve covariate balance within subgroups, but it also increases variance. We propose a new method, the subgroup balancing propensity score, to ensure good subgroup balance as well as to control the variance inflation. For each subgroup, the subgroup balancing propensity score chooses to use either the overall sample or the subgroup (sub)sample to estimate the propensity scores for the units within that subgroup, in order to optimize a criterion accounting for a set of covariate-balancing moment conditions for both the overall sample and the subgroup samples. We develop two versions of subgroup balancing propensity score corresponding to matching and weighting, respectively. We devise a stochastic search algorithm to estimate the subgroup balancing propensity score when the number of subgroups is large. We demonstrate through simulations that the subgroup balancing propensity score improves the performance of propensity score methods in estimating subgroup treatment effects. We apply the subgroup balancing propensity score method to the Italy Survey of Household Income and Wealth (SHIW) to estimate the causal effects of having debit card on household consumption for different income groups.

Controlling for confounding via propensity score methods can result in biased estimation of the conditional AUC: A simulation study.

In the medical literature, there has been an increased interest in evaluating association between exposure and outcomes using nonrandomized observational studies. However, because assignments to exposure are not random in observational studies, comparisons of outcomes between exposed and nonexposed subjects must account for the effect of confounders. Propensity score methods have been widely used to control for confounding, when estimating exposure effect. Previous studies have shown that conditioning on the propensity score results in biased estimation of conditional odds ratio and hazard ratio. However, research is lacking on the performance of propensity score methods for covariate adjustment when estimating the area under the ROC curve (AUC). In this paper, AUC is proposed as measure of effect when outcomes are continuous. The AUC is interpreted as the probability that a randomly selected nonexposed subject has a better response than a randomly selected exposed subject. A series of simulations has been conducted to examine the performance of propensity score methods when association between exposure and outcomes is quantified by AUC; this includes determining the optimal choice of variables for the propensity score models. Additionally, the propensity score approach is compared with that of the conventional regression approach to adjust for covariates with the AUC. The choice of the best estimator depends on bias, relative bias, and root mean squared error. Finally, an example looking at the relationship of depression/anxiety and pain intensity in people with sickle cell disease is used to illustrate the estimation of the adjusted AUC using the proposed approaches.

Performance study of propensity score methods against regression with covariate adjustment

Abstract In observational study, handling confounders is a primary issue in measuring treat-ment e ect of interest. Historically, a regression with covariate adjustment (covariate-adjusted regression) has been the typical approach to estimate treatment e ect incor-porating potential confounders into model. However, ever since the introduction of thepropensity score, covariate-adjusted regression has been gradually replaced in medicalliteratures with various balancing methods based on propensity score. On the otherhand, there is only a paucity of researches assessing propensity score methods com-pared with the covariate-adjusted regression. This paper examined the performance ofpropensity score methods in estimating risk di erence and compare their performancewith the covariate-adjusted regression by a Monte Carlo study. The study demonstratedin general the covariate-adjusted regression with variable selection procedure outper-formed propensity-score-based methods in terms both of bias and MSE, suggesting thatthe classical regression method needs to be considered, rather than the propensity scoremethods, if a performance is a primary concern.Keywords: Inverse probability of treatment weighting, Monte Carlo study, propensityscore.

A1-3: The Performance of Propensity Score Methods for Estimating Hazard Ratio in Time-to-Event Data

Background/Aims Propensity Score (PS) methods have been used increasingly in observational studies to reduce bias due to confounding. The performance of PS for estimating hazard ratios (HR) has not been systematically studied. Monte Carlo simulations were performed to evaluate four methods of utilizing PS: 1:1 matching, stratification, PS adjustment, and inverse probability weighting (IPW).

Prognostic score–based balance measures can be a useful diagnostic for propensity score methods in comparative effectiveness research

ObjectiveExamining covariate balance is the prescribed method for determining the degree to which propensity score methods should be successful at reducing bias. This study assessed the performance of various balance measures, including a proposed balance measure based on the prognostic score (similar to a disease risk score), to determine which balance measures best correlate with bias in the treatment effect estimate. Study Design and SettingThe correlations of multiple common balance measures with bias in the treatment effect estimate produced by weighting by the odds, subclassification on the propensity score, and full matching on the propensity score were calculated. Simulated data were used, based on realistic data settings. Settings included both continuous and binary covariates and continuous covariates only. ResultsThe absolute standardized mean difference (ASMD) in prognostic scores, the mean ASMD (in covariates), and the mean t-statistic all had high correlations with bias in the effect estimate. Overall, prognostic scores displayed the highest correlations with bias of all the balance measures considered. Prognostic score measure performance was generally not affected by model misspecification, and the prognostic score measure performed well under a variety of scenarios. ConclusionResearchers should consider using prognostic score–based balance measures for assessing the performance of propensity score methods for reducing bias in nonexperimental studies.

CA5-02: Assessing the Performance of Propensity Score Methods for Estimating the Probability of Receiving Cervical Procedures

Background/Aims Propensity scores provide an alternative to traditional regression modeling for assessing the effect of exposure on an outcome in the presence of confounders. We describe methods to establish propensity scores for a cohort of Kaiser Permanente Northwest (KPNW) members receiving surgical procedures of the cervix.

Performance of propensity score methods when comparison groups originate from different data sources

To examine the performance of propensity score-based methods for estimating relative risks when exposed and comparison subjects are selected from different data sources. We conducted Monte Carlo simulations to assess the performance of propensity score methods under various scenarios in which exposed and comparison subjects were selected from different data sources for a comparative effectiveness study of a medical device. The use of propensity score methods in our simulated data scenarios often yielded estimates of relative risk that were close to the true effect, unless the comparison group differed from the exposed group systematically on a factor associated with the outcome. This situation caused severe bias regardless of which method was used but could be overcome if the exposed group could be restricted similarly to the comparison group. Mean square error of relative risk estimates was lowest for similarly restricted study groups and when the comparison group could be considered a random sample of the source population that generated the exposed group. When exposed and comparison groups originated from different data sources, all propensity score methods yielded relatively unbiased and consistent estimates of relative risk in most situations reflected in our simulation study.

The performance of different propensity score methods for estimating marginal odds ratios

The propensity score which is the probability of exposure to a specific treatment conditional on observed variables. Conditioning on the propensity score results in unbiased estimation of the expected difference in observed responses to two treatments. In the medical literature, propensity score methods are frequently used for estimating odds ratios. The performance of propensity score methods for estimating marginal odds ratios has not been studied. We performed a series of Monte Carlo simulations to assess the performance of propensity score matching, stratifying on the propensity score, and covariate adjustment using the propensity score to estimate marginal odds ratios. We assessed bias, precision, and mean-squared error (MSE) of the propensity score estimators, in addition to the proportion of bias eliminated due to conditioning on the propensity score. When the true marginal odds ratio was one, then matching on the propensity score and covariate adjustment using the propensity score resulted in unbiased estimation of the true treatment effect, whereas stratification on the propensity score resulted in minor bias in estimating the true marginal odds ratio. When the true marginal odds ratio ranged from 2 to 10, then matching on the propensity score resulted in the least bias, with a relative biases ranging from 2.3 to 13.3 per cent. Stratifying on the propensity score resulted in moderate bias, with relative biases ranging from 15.8 to 59.2 per cent. For both methods, relative bias was proportional to the true odds ratio. Finally, matching on the propensity score tended to result in estimators with the lowest MSE.