Posttreatment Bias Research Articles

In Search of the Causes of the Globalization Backlash: Methodological Considerations on Post-treatment Bias

We study the implications of post-treatment bias in the context of the globalization backlash. We discuss whether horse-race regressions can inform about the relative role of economic versus cultural drivers. We make three methodological points: (1) if and insofar as cultural variables are post-treatment with respect to economic factors, the estimates of the effect of economic shocks on voting are biased in regressions that include cultural controls (and vice versa); (2) for the same reason, such horse-race regressions do not allow to accurately estimate the relative role of economic versus cultural factors; (3) one cannot infer mediation effects from changes in regression coefficients for a given factor of interest before and after including post-treatment controls. We accompany the methodological discussion with empirical evidence on the relevance of post-treatment bias in studies of the globalization backlash, both by replicating and expanding on earlier studies, and by presenting novel cross-country results on the culture-economy nexus.

(Empirical) Bayes Approaches to Parallel Trends

We consider Bayes and Empirical Bayes (EB) approaches for dealing with violations of parallel trends. In the Bayes approach, the researcher specifies a prior over both the pretreatment violations of parallel trends delta-pre and the posttreatment violations delta-post. The researcher then updates their posterior about the posttreatment bias delta-post given an estimate of the pre-trends delta-pre. This allows them to form posterior means and credible sets for the treatment effect of interest, Delta-post. In the EB approach, the prior on the violations of parallel trends is learned from the pretreatment observations. We illustrate these approaches in two empirical applications.

Causal inference for psychologists who think that causal inference is not for them

AbstractCorrelation does not imply causation and psychologists' causal inference training often focuses on the conclusion that therefore experiments are needed—without much consideration for the causal inference frameworks used elsewhere. This leaves researchers ill‐equipped to solve inferential problems that they encounter in their work, leading to mistaken conclusions and incoherent statistical analyses. For a more systematic approach to causal inference, this article provides brief introductions to the potential outcomes framework—the “lingua franca” of causal inference—and to directed acyclic graphs, a graphical notation that makes it easier to systematically reason about complex causal situations. I then discuss two issues that may be of interest to researchers in social and personality psychology who think that formalized causal inference is of little relevance to their work. First, posttreatment bias: In various common scenarios (noncompliance, mediation analysis, missing data), researchers may analyze data from experimental studies in a manner that results in internally invalid conclusions, despite randomization. Second, tests of incremental validity: Routine practices in personality psychology suggest that they may be conducted for at least two different reasons (to demonstrate the non‐redundancy of new scales, to support causal conclusions) without being particularly suited for either purpose. Taking causal inference seriously is challenging; it reveals assumptions that may make many uncomfortable. However, ultimately it is a necessary step to ensure the validity of psychological research.

Fixed Effects and Post-Treatment Bias in Legacy Studies

Pepinsky, Goodman, and Ziller (2024, American Political Science Review, PGZ) reassess a recent study on the long-term consequences of concentration camps in Germany. The authors conclude that accounting for contemporary (i.e., post-treatment) state heterogeneity in the models provides unbiased estimates of the effects of camps on current-day outgroup intolerance. In this note, we show that PGZ’s empirical strategy rests on (a) a mischaracterization of what regional fixed effects capture and (b) two unrealistic assumptions that can be avoided with pre-treatment state fixed effects. We further demonstrate that results from the original article remain substantively the same when we incorporate regional fixed effects correctly. Finally, simulations reveal that camp proximity consistently outperforms spatially correlated noise in this specific study. The note contributes to the growing literature on legacy studies by advancing the discussion about the correct modeling choices in this challenging field.

No Evidence that Measuring Moderators Alters Treatment Effects

AbstractSocial scientists are frequently interested in who is most responsive to a treatment. By necessity, such moderation experiments often rely on observed moderators, such as partisan identity. These designs have led to an ongoing debate about where to measure moderators—immediately prior to the treatment, after the treatment, or in a prior wave of a panel survey. Measuring a moderator prior to the treatment is the most efficient and avoids posttreatment bias, but it raises concerns about priming. We contribute to this debate by systematically studying whether measuring moderators prior to an experiment affects the results. Across six different experiments, each involving a commonly used moderator, we find little evidence of priming effects, even when a moderator is placed immediately before the experiment. Our findings thus help resolve the debate, suggesting that researchers should measure moderators pretreatment. We conclude with advice on designing well‐powered moderation experiments.

Modeling Spatial Heterogeneity and Historical Persistence: Nazi Concentration Camps and Contemporary Intolerance

A wealth of recent research in comparative politics examines how spatial variation in historical conditions shapes modern political outcomes. In an article in the American Political Science Review, Homola, Pereira, and Tavits argue that Germans who live nearer to former Nazi concentration camps are more likely to display out-group intolerance. Clarifying the conceptual foundations of posttreatment bias and reviewing the historical record on postwar state creation in Germany, we argue that state-level differences confound the relationship between distance to camps and out-group intolerance. Using publicly available European Values Survey data and electoral results from 2017, we find no consistent evidence that distance to camps is related to contemporary values. Our findings have implications for literatures on historical persistence, causal inference with spatial data, Holocaust studies, and outgroup tolerance.

How effective is reducing traffic speed for safer work zones? Methodology and a case study in Pennsylvania

Transportation agencies post and enforce reduced speed limits in work zones to ensure work zone safety, since traffic speed is found to be associated with work zone crash risks. However, prior findings on the relationship between speed and crash rate in work zones are inconsistent. This may be attributed to the methods of statistical associations between traffic speed and crash risks that do not necessarily discover true causal relations. In fact, work zone presence could lead to the reduction of actual traffic speed that influences crash risks, where it may also directly impose effects on crash risks as a result of work zone configurations. The actual traffic speed (not posted speed limit) is also known as a “mediator” where work zones can indirectly impact the crash risks. It is challenging to rigorously separate the causal effect of traffic speed on work zone crash risk from that directly caused by work zones. The underlying causal relation could help to determine what reduced post speed limit (with enforcement) is necessary to ensure work zone safety under the most desired “actual traffic speed”. This study proposes to use the sequential g-estimation and the regression discontinuity design to estimate the controlled direct effect of traffic speed on work zone crashes. Two research gaps are identified and filled: inaccurate inferences of the effect of reduced speed limit in work zones as a result of ignoring (1) potential post-treatment bias since traffic speed is a mediator; and (2) potential confounding bias caused by unobservable roadway characteristics. The proposed methodology was applied to 4008 work zones in Pennsylvania from 2015 to 2017, and the results were validated through a series of robustness tests. The results indicate that the direct causal effect of the presence of work zones on crash risk is significantly positive when the traffic speed is relatively low (i.e., lower than 55 mph in this case study), while traffic speed has a positive causal effect on crash occurrences when the actual traffic speed is high (i.e., greater or equal to 55 mph). It suggests that strictly enforcing reduced posted speed limits in work zones is particularly effective when the actual traffic speed is greater than 55 mph. This is particularly true on roadways with high traffic volume (i.e., AADT > 20,000 vehicles per day), long, and daytime work zones (i.e., > 3000 m). On the other hand, the effect of enforcing reduced speed on work zone safety is unclear when the actual speed is already low. In this case, improving work zone configurations and driving behaviors may be more effective in reducing crash risks.

Can Close Election Regression Discontinuity Designs Identify Effects of Winning Politician Characteristics?

AbstractPolitician characteristic regression discontinuity (PCRD) designs leveraging close elections are widely used to isolate effects of an elected politician characteristic on downstream outcomes. Unlike standard regression discontinuity designs, treatment is defined by a predetermined characteristic that could affect a politician's victory margin. I prove that, by conditioning on politicians who win close elections, PCRD estimators identify the effect of the specific characteristic of interest and all compensating differentials—candidate‐level characteristics that ensure elections remain close between candidates who differ in the characteristic of interest. Avoiding this asymptotic bias generally requires assuming either that the characteristic of interest does not affect candidate vote shares or that no compensating differential affects the outcome. Because theories of voting behavior suggest that neither strong assumption usually holds, I further analyze the implications for interpreting continuity tests and consider if and how covariate adjustment, bounding, and recharacterizing treatment can mitigate the posttreatment bias afflicting PCRD designs.

Moderator Placement in Survey Experiments: Racial Resentment and the “Welfare” versus “Assistance to the Poor” Question Wording Experiment

AbstractResearchers face difficult decisions about whether to ask potential moderators before or after a survey experiment. Competing concerns exist about priming respondents before the experiment and about introducing post-treatment bias. We replicate the classic “welfare” versus “assistance to the poor” survey experiment, randomly assigning respondents to be asked a battery of racial resentment questions either before or after the question wording experiment. We find little evidence that the question wording effect is different between those who are asked about racial resentment before versus after the experiment. Furthermore, we find little evidence that measured racial resentment is affected by this ordering or by the question wording treatment itself.

A Causal Framework for Observational Studies of Discrimination

In studies of discrimination, researchers often seek to estimate a causal effect of race or gender on outcomes. For example, in the criminal justice context, one might ask whether arrested individuals would have been subsequently charged or convicted had they been a different race. It has long been known that such counterfactual questions face measurement challenges related to omitted-variable bias, and conceptual challenges related to the definition of causal estimands for largely immutable characteristics. Another concern, which has been the subject of recent debates, is post-treatment bias: many studies of discrimination condition on apparently intermediate outcomes, like being arrested, that themselves may be the product of discrimination, potentially corrupting statistical estimates. There is, however, reason to be optimistic. By carefully defining the estimand—and by considering the precise timing of events—we show that a primary causal quantity of interest in discrimination studies can be estimated under an ignorability condition that may hold approximately in some observational settings. We illustrate these ideas by analyzing both simulated data and the charging decisions of a prosecutor’s office in a large county in the United States.

Telescope Matching for Reducing Model Dependence in the Estimation of the Effects of Time-Varying Treatments: An Application to Negative Advertising

Abstract Time-varying treatments are prevalent in the social sciences. For example, a political campaign might decide to air attack ads against an opponent, but this decision to go negative will impact polling and, thus, future campaign strategy. If an analyst naively applies methods for point exposures to estimate the effect of earlier treatments, this would lead to post-treatment bias. Several existing methods can adjust for this type of time-varying confounding, but they typically rely on strong modelling assumptions. In this paper, we propose a novel two-step matching procedure for estimating the effect of two-period treatments. This method, telescope matching, reduces model dependence without inducing post-treatment bias by using matching with replacement to impute missing counterfactual outcomes. It then employs flexible regression models to correct for bias induced by imperfect matches. We derive the asymptotic properties of the telescope matching estimator and provide a consistent estimator for its variance. We illustrate telescope matching by investigating the effect of negative campaigning in US Senate and gubernatorial elections. Using the method, we uncover a positive effect on turnout of negative ads early in a campaign and a negative effect of early negativity on vote shares.

Measuring Supreme Court Case Complexity

Case complexity is central to the study of judicial politics. The dominant measures of Supreme Court case complexity use information on legal issues and provisions observed post-decision. As a result, scholars using these measures to study merits stage outcomes such as bargaining, voting, separate opinion production, and opinion content introduce post-treatment bias and exacerbate endogeneity concerns. Furthermore, existing issue measures are not valid proxies for complexity. Leveraging information on issues and provisions extracted from merits briefs, we develop a new latent measure of Supreme Court case complexity. This measure maps with the prevailing understanding of the underlying concept while mitigating inferential threats that hamper empirical evaluations. Our brief-based measurement strategy is generalizable to other contexts where it is important to generate exogenous and pre-treatment indicators for use in explaining merits decisions.

Does Free-Market Reform Induce Protest? Selection, Post-Treatment Bias, and Depoliticization

AbstractAcross myriad literatures, it is widely held that expanding economic grievances induce violence, protest, or other forms of backlash. In Latin America, where economic liberalization deepened the downturn of the ‘lost decade’ of the 1980s (and 1990s), reform has been tightly associated with protest and mobilization. At the same time, liberal economic reforms have proven to be remarkably durable, even where long-promised benefits are hard to discern. This article makes the case that economically liberal reforms, despite inducing or deepening severe and sustained economic downturns, have actually undermined political protest. Previous work confirming the conventional wisdom foundered on two main methodological problems. First, selection into economic reform was a consequence of the very economic pain and macroeconomic imbalances it also served to induce. Secondly, because of this, these key (macro)economic characteristics are both pre- and post-treatment. Utilizing a marginal structural model approach to assess the impact of economic liberalization on protest outcomes net of this selection process, and the prior history of treatment, the study finds that painful reform reduces political protest even as it heightens grievances. This depoliticizing dynamic helps to explain the surprising durability of liberal reforms in Latin America.

Measuring Supreme Court Case Complexity

Abstract Case complexity is central to the study of judicial politics. The dominant measures of Supreme Court case complexity use information on legal issues and provisions observed postdecision. As a result, scholars using these measures to study merits stage outcomes such as bargaining, voting, separate opinion production, and opinion content introduce posttreatment bias and exacerbate endogeneity concerns. Furthermore, existing issue measures are not valid proxies for complexity. Leveraging information on issues and provisions extracted from merits briefs, we develop a new latent measure of Supreme Court case complexity. This measure maps with the prevailing understanding of the underlying concept while mitigating inferential threats that hamper empirical evaluations. Our brief-based measurement strategy is generalizable to other contexts where it is important to generate exogenous and pretreatment indicators for use in explaining merits decisions. (JEL K00, K40)

Studying Identities with Experiments: Weighing the Risk of Posttreatment Bias Against Priming Effects – Corrigendum

An abstract is not available for this content. As you have access to this content, full HTML content is provided on this page. A PDF of this content is also available in through the ‘Save PDF’ action button.

Studying Identities with Experiments: Weighing the Risk of Posttreatment Bias Against Priming Effects

AbstractScholars from across the social sciences argue that identities – such as race, ethnicity, and gender – are highly influential over individuals’ attitudes, actions, and evaluations. Experiments are becoming particularly integral for allowing identity scholars to explain how these social attachments shape our political behavior. In this letter, we draw attention to how identity scholars should approach the common practice of assessing moderators, measuring control variables, and detecting effect heterogeneity using covariates. Special care must be taken when deciding where to place measures of demographic covariates in identity-related experiments, as these cases pose unique challenges from how scholars traditionally approach experimental design. We argue in this letter that identity scholars, particularly those whose subjects identify as women or minorities, are often right to measure covariates of interest posttreatment.

RNA oxidation and iron levels in patients with diabetes

AimThe urinary biomarker for oxidative stress to RNA, 8-oxo-7,8-dihydro-guanosine (8-oxoGuo) is associated with mortality in patients with type 2 diabetes. Iron has also been linked to diabetes. In individuals with untreated hereditary iron overload it has been observed that 8-oxoGuo was higher compared to controls. In the current study, we hypothesized that 8-oxoGuo was associated with diagnosis of diabetes, and that iron confounded this association. MethodsParticipants from a general Danish population were included in the study (n = 3567). UPLC-MS/MS method was used for 8-oxoGuo (nmol/mmol creatinine) measurement in spot urine. Iron biomarkers included total plasma iron, ferritin, transferrin saturation (TS) and transferrin. Results8-oxoGuo was 17% higher in diabetes patients (n = 208) compared to non-diabetes controls. Unadjusted logistic regression model showed an odds ratio of diabetes of 1.38 (95%CI:1.21–1.57, P < 0.0001) per unit increase of 8-oxoGuo. When the model was adjusted for possible confounders the odds ratio was 1.09 (95%CI:0.94–1.26, P = 0.24). When additional adjustment was performed including ferritin, TS, or transferrin, respectively, the OR were 1.14 (95%CI:0.97–1.33, P = 0.09), 1.10 (95%CI: 0.95–1.28, P = 0.18), and 1.17 (95%CI:1.01–1.38, P = 0.04). ConclusionsOur study indicates that 8-oxoGuo is higher in diabetes patients. The lack of association between 8-oxoGuo and diabetes in the adjusted model may be due to the cross-sectional design including post-treatment bias. Our data did not show consistent effect of all iron biomarkers in relation to diabetes. Most likely, the iron biomarkers were affected by inflammation thus not reflecting true iron levels.

Avoiding Post-Treatment Bias in Audit Experiments

Audit experiments are used to measure discrimination in a large number of domains (Employment: Bertrand et al. (2004); Legislator responsiveness: Butler et al. (2011); Housing: Fang et al. (2018)). Audit studies all have in common that they estimate the average difference in response rates depending on randomly varied characteristics (such as the race or gender) of a requester. Scholars conducting audit experiments often seek to extend their analyses beyond the effect on response to the effects on the quality of the response. Response is a consequence of treatment; answering these important questions well is complicated by post-treatment bias (Montgomery et al., 2018). In this note, I consider a common form of post-treatment bias that occurs in audit experiments.

How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It

AbstractIn principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for posttreatment variables in statistical models, eliminating observations based on posttreatment criteria, or subsetting the data based on posttreatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on posttreatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental posttreatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real‐world data. We conclude by providing applied researchers with recommendations for best practice.

Validity of new methods to evaluate renal function in cancer patients treated with cisplatin.

PurposeCreatinine clearance (Ccr) is used as a marker of renal function in cancer chemotherapy, but it is not correlated with glomerular filtration rate (GFR) after high-dose cisplatin treatment. In addition to Ccr, measured using 24-h urine collection (24-h Ccr) or Cockcroft–Gault formula (CGF), the Chronic Kidney Disease Epidemiology Collaboration (CKD–EPI) equation and the Japanese GFR estimation equation (the Japanese equation) have been recently developed to estimate GFR for predicting renal function. However, these equations remain to be evaluated, particularly in cancer patients treated with cisplatin. Therefore, we investigated the validity of these equations for predicting the GFR in cancer patients treated with cisplatin.MethodsGFR was measured by inulin clearance (Cin) in 50 cancer patients and compared with GFR estimated by the CKD–EPI equation, the Japanese equation, and Ccr estimated by CGF or measured by 24-h Ccr before the first and third cisplatin-containing chemotherapy cycles (considered pretreatment and posttreatment, respectively).ResultsBefore treatment, the CKD–EPI and the Japanese equations estimated GFR with higher accuracy than Ccr. Posttreatment bias values for GFR estimation using the CKD–EPI and the Japanese equations were lower than those for Ccr. The CKD–EPI and the Japanese equations were also more precise than Ccr. However, for patients with low renal function, these equations still overestimated Cin.ConclusionThe CKD–EPI and the Japanese equations estimated GFR with lower bias and higher precision than Ccr pre- and postcisplatin treatment. This study is registered at UMIN: 000002167.