Plausible Counterfactuals Research Articles

Are there lasting impacts of China’s anti-poverty program? A synthetic control analysis

ABSTRACT China’s National Poor Counties (NPCs) program represents one of the largest anti-poverty efforts ever undertaken in a developing country. In spite of this, no causal analysis has been undertaken to examine whether regions in poverty at the end of the program actually achieve sustainable growth after support is removed. This paper provides the first long-term analysis of China’s NPCs (National Poor Counties) program on local development, investigating whether poor regions continue to develop economically after exiting the anti-poverty program. Focusing on the period 1986 to 2010, I construct plausible counterfactuals for counties that participated and graduated from the NPCs registry using the synthetic control method. This exercise reveals several new insights. First, on average, graduates of China’s anti-poverty program exhibit roughly the same growth outcomes as those still receiving support from the program. Second, I show that focusing on the average impact masks sizable heterogeneity when disaggregating to a case by case analysis of counties. Importantly, for the subset of counties which graduated from the first wave of the poverty reduction program in 1993, there is a demonstrably faster expansion post-graduation. For more recent county’s graduating from the program, however, I find significantly lower income growth of NPCs program completion in the long run.

Challenges in Evaluating a Community-Level Intervention to Address Root Causes of Youth Violence

Violence disproportionately impacts Black American youth, representing a major health disparity. Addressing the possible root causes of structural inequities to reduce violence may increase the impact of prevention strategies. However, efforts to evaluate the impact of such interventions pose numerous methodological challenges, particularly around selecting an effective evaluation design to detect change at the community level, with adequate power and sampling, and appropriate constructs and measurement strategies. We propose a multiple baseline experimental design to evaluate the impact of a community-level youth violence and suicidality prevention strategy. A multiple baseline experimental design with multiple community units balances the need for scientific rigor with practical and values-based considerations. It includes randomization and plausible counterfactuals without requiring large samples or placing some communities in the position of not receiving the intervention. Considerations related to the conceptualization of the logic model, mechanisms of change, and health disparity outcomes informed the development of the measurement strategy. The strengths and weaknesses of a multiple baseline experimental design are discussed in comparison to versions of randomized clinical trials. Future health disparity intervention evaluation research will benefit from (1) building a shared sense of urgent public need to promote health; (2) respecting the validity of values- and partnership-based decision-making; and (3) promoting community-based and systems-level partnerships in scientific grant funding. The described study has been registered prospectively at clinicaltrials.gov, Protocol Record 21–454.

Evaluation of averted loss gains under Victorian biodiversity offset policy

AbstractBiodiversity offsetting seeks to balance losses in biodiversity from development with offset actions that generate biodiversity gains. In Victoria, Australia, native vegetation offset policy aims to achieve No Net Loss (NNL) in biodiversity, meaning gains from offsetting must be at least equivalent to development losses. Gains may be generated from measures that protect the existing biodiversity from loss (averted loss gains), and those that restore or enhance the biodiversity (improvement gains). However, previous work has suggested the Victorian offset policy overestimates averted loss gains, allowing a larger development impact for the same offset. Here, we undertake a detailed examination of five impact‐offset exchanges permitted under Victorian offset policy in 2019 and 2020, comparing gains under the policy to those resulting from plausible counterfactuals. We find that the projected gains from averted loss were likely to be overestimated by between 3 and 240 times, compared to the counterfactuals. If this level of over‐estimation holds throughout the scheme, it would mean that up to 86% of offsets purchased do not compensate for permitted impacts to native vegetation, thwarting Victoria's NNL policy objective. We discuss the key policy issues and make recommendations to address over‐estimating averted loss gains.

Retail Electricity Market Restructuring and Retail Rates

Prior to the 1990s, all U.S. states used a “cost of service (COS)” regulation regime in which investor-owned utilities were allowed to recover prudently incurred costs plus a rate of return on capital expenditures, and retail customers were unable to choose their electricity supplier. From 1996–2000, multiple states passed retail electricity market “restructuring.” This empirical research examines the effect of retail restructuring on electricity prices to final consumers. We find that rates increased in restructured states relative to plausible counterfactuals in the years post-restructuring. But by twelve years after retail restructuring, we no longer observe any difference. We investigate plausible mechanisms, finding evidence that retail prices became more responsive to natural gas prices due to retail restructuring, the timing of which coincided with increases in natural gas prices nationally. We also test for whether restructuring had distributional effects across customer classes and find that in the short run residential customers benefited relative to industrial customers during transition periods, but that this difference does not persist into full implementation.

Explaining holistic image regressors and classifiers in urban analytics with plausible counterfactuals

We propose a new form of plausible counterfactual explanation designed to explain the behaviour of computer vision systems used in urban analytics that make predictions based on properties across the entire image, rather than specific regions of it. We illustrate the merits of our approach by explaining computer vision models used to analyse street imagery, which are now widely used in GeoAI and urban analytics. Such explanations are important in urban analytics as researchers and practioners are increasingly reliant on it for decision making. Finally, we perform a user study that demonstrate our approach can be used by non-expert users, who might not be machine learning experts, to be more confident and to better understand the behaviour of image-based classifiers/regressors for street view analysis. Furthermore, the method can potentially be used as an engagement tool to visualise how public spaces can plausibly look like. The limited realism of the counterfactuals is a concern which we hope to improve in the future.

On Generating Plausible Counterfactual and Semi-Factual Explanations for Deep Learning

There is a growing concern that the recent progress made in AI, especially regarding the predictive competence of deep learning models, will be undermined by a failure to properly explain their operation and outputs. In response to this disquiet, counterfactual explanations have become very popular in eXplainable AI (XAI) due to their asserted computational, psychological, and legal benefits. In contrast however, semi-factuals (which appear to be equally useful) have surprisingly received no attention. Most counterfactual methods address tabular rather than image data, partly because the non-discrete nature of images makes good counterfactuals difficult to define; indeed, generating plausible counterfactual images which lie on the data manifold is also problematic. This paper advances a novel method for generating plausible counterfactuals and semi-factuals for black-box CNN classifiers doing computer vision. The present method, called PlausIble Exceptionality-based Contrastive Explanations (PIECE), modifies all “exceptional” features in a test image to be “normal” from the perspective of the counterfactual class, to generate plausible counterfactual images. Two controlled experiments compare this method to others in the literature, showing that PIECE generates highly plausible counterfactuals (and the best semi-factuals) on several benchmark measures.

Does constitutional entrenchment matter for economic freedom?

AbstractA growing number of studies explore the determinants of economic freedom. Very few of them consider constitutional design. We study entrenchment, that is, the extent to which constitutions are more costly to change than ordinary policies and institutions. We utilize 1970–2017 data and study episodes where countries adopted meaningfully more entrenched constitutions. Using matching methods, we construct plausible counterfactuals against which to compare their post‐treatment changes in economic freedom. We report no significant effects on overall freedom. There is some evidence that entrenchment leads to smaller government size, more regulation, and weaker property rights. However, none of these results are robust.

Does Constitutional Entrenchment Matter for Economic Freedom?

A growing number of studies explore the determinants of economic freedom. Very few of them consider constitutional design. We study entrenchment, i.e., the extent to which constitutions are more costly to change than ordinary policies and institutions. We utilize 1970-2017 data and study episodes where countries adopted meaningfully more entrenched constitutions. Using matching methods, we construct plausible counterfactuals against which to compare their post-treatment changes in economic freedom. We report no significant effects on overall freedom. There is some evidence that entrenchment leads to smaller government size, more regulation, and weaker property rights. However, none of these results are robust.

The specter of irreparable ignorance: counterfactuals and causality in economics.

Those economists who have emphasized true uncertainty have tended to draw an epistemic distinction between an ascertainable past and an unknowable future. But in one critical respect—in extracting causal relationships—that epistemic distinction is not warranted. Whether they are situated in the past or future, causal arguments in economics depend equally on counterfactual reasoning. Counterfactualizing entails the construction of fictitious narratives—narratives about worlds that do not exist. Unfortunately, there is no dependable method for ascertaining the uniquely correct counterfactual. This implies that causal claims in economics, too, are irreducibly fictitious. The chief value of counterfactuals, then, is not to prove causation but to help scholars and practitioners confront an inscrutable world—to imagine and prepare for unknowable possible futures. In this endeavor, economic pluralism, which expands the range of plausible counterfactuals, is to be taken as a virtue rather than a curse.

Reconciling multiple counterfactuals when evaluating biodiversity conservation impact in social-ecological systems.

When evaluating the impact of a biodiversity conservation intervention, a counterfactual is typically needed. Counterfactuals are possible alternative system trajectories in the absence of an intervention. Comparing observed outcomes against the chosen counterfactual allows the impact (change attributable to the intervention) to be determined. Because counterfactuals by definition never occur, they must be estimated. Sometimes, there may be many plausible counterfactuals, including various drivers of biodiversity change and defined on a range of spatial or temporal scales. Here, we posit that, by definition, conservation interventions always take place in social-ecological systems (SES) (i.e., ecological systems integrated with human actors). Evaluating the impact of an intervention in an SES, therefore, means taking into account the counterfactuals assumed by different human actors. Use of different counterfactuals by different actors will give rise to perceived differences in the impacts of interventions, which may lead to disagreement about its success or the effectiveness of the underlying approach. Despite that there are biophysical biodiversity trends, it is often true that no single counterfactual is definitively the right one for conservation assessment, so multiple evaluations of intervention efficacy could be considered justifiable. Therefore, we propose calculating the sum of perceived differences, which captures the range of impact estimates associated with different actors in a given SES. The sum of perceived differences gives some indication of how closely actors in an SES agree on the impacts of an intervention. We applied the concept of perceived differences to a set of global, national, and regional case studies (e.g., global realization of Aichi Target 11 for marine protected areas, effect of biodiversity offsetting on vegetation condition in Australia, and influence of conservation measures on an endangered ungulate in Central Asia). We explored approaches for minimizing the sum, including a combination of negotiation and structured decision making, careful alignment of expectations on scope and measurement, and explicit recognition of any intractable differences between stakeholders.

What if? Neural activity underlying semantic and episodic counterfactual thinking

Counterfactual thinking (CFT) is the process of mentally simulating alternative versions of known facts. In the past decade, cognitive neuroscientists have begun to uncover the neural underpinnings of CFT, particularly episodic CFT (eCFT), which activates regions in the default network (DN) also activated by episodic memory (eM) recall. However, the engagement of DN regions is different for distinct kinds of eCFT. More plausible counterfactuals and counterfactuals about oneself show stronger activity in DN regions compared to implausible and other- or object-focused counterfactuals. The current study sought to identify a source for this difference in DN activity. Specifically, self-focused counterfactuals may also be more plausible, suggesting that DN core regions are sensitive to the plausibility of a simulation. On the other hand, plausible and self-focused counterfactuals may involve more episodic information than implausible and other-focused counterfactuals, which would imply DN sensitivity to episodic information. In the current study, we compared episodic and semantic counterfactuals generated to be plausible or implausible against episodic and semantic memory reactivation using fMRI. Taking multivariate and univariate approaches, we found that the DN is engaged more during episodic simulations, including eM and all eCFT, than during semantic simulations. Semantic simulations engaged more inferior temporal and lateral occipital regions. The only region that showed strong plausibility effects was the hippocampus, which was significantly engaged for implausible CFT but not for plausible CFT, suggestive of binding more disparate information. Consequences of these findings for the cognitive neuroscience of mental simulation are discussed.

Improving the Interpretation of Fixed Effects Regression Results

Fixed effects estimators are frequently used to limit selection bias. For example, it is well known that with panel data, fixed effects models eliminate time-invariant confounding, estimating an independent variable’s effect using only within-unit variation. When researchers interpret the results of fixed effects models, they should therefore consider hypothetical changes in the independent variable (counterfactuals) that could plausibly occur within units to avoid overstating the substantive importance of the variable’s effect. In this article, we replicate several recent studies which used fixed effects estimators to show how descriptions of the substantive significance of results can be improved by precisely characterizing the variation being studied and presenting plausible counterfactuals. We provide a checklist for the interpretation of fixed effects regression results to help avoid these interpretative pitfalls.

The logic of counterfactual analysis in case-study explanation.

In this paper, we develop a set-theoretic and possible worlds approach to counterfactual analysis in case-study explanation. Using this approach, we first consider four kinds of counterfactuals: necessary condition counterfactuals, SUIN condition counterfactuals, sufficient condition counterfactuals, and INUS condition counterfactuals. We explore the distinctive causal claims entailed in each, and conclude that necessary condition and SUIN condition counterfactuals are the most useful types for hypothesis assessment in case-study research. We then turn attention to the development of a rigorous understanding of the 'minimal-rewrite' rule, linking this rule to insights from set theory about the relative importance of necessary conditions. We show why, logically speaking, a comparative analysis of two necessary condition counterfactuals will tend to favour small events and contingent happenings. A third section then presents new tools for specifying the level of generality of the events in a counterfactual. We show why and how the goals of formulating empirically important versus empirically plausible counterfactuals stand in tension with one another. Finally, we use our framework to link counterfactual analysis to causal sequences, which in turn provides advantages for conducting counterfactual projections.

Estimating the Cost-Effectiveness of HIV Prevention Programmes in Vietnam, 2006-2010: A Modelling Study.

IntroductionVietnam has been largely reliant on international support in its HIV response. Over 2006-2010, a total of US$480 million was invested in its HIV programmes, more than 70% of which came from international sources. This study investigates the potential epidemiological impacts of these programmes and their cost-effectiveness.MethodsWe conducted a data synthesis of HIV programming, spending, epidemiological, and clinical outcomes. Counterfactual scenarios were defined based on assumed programme coverage and behaviours had the programmes not been implemented. An epidemiological model, calibrated to reflect the actual epidemiological trends, was used to estimate plausible ranges of programme impacts. The model was then used to estimate the costs per averted infection, death, and disability adjusted life-year (DALY).ResultsBased on observed prevalence reductions amongst most population groups, and plausible counterfactuals, modelling suggested that antiretroviral therapy (ART) and prevention programmes over 2006-2010 have averted an estimated 50,600 [95% uncertainty bound: 36,300–68,900] new infections and 42,600 [36,100–54,100] deaths, resulting in 401,600 [312,200–496,300] fewer DALYs across all population groups. HIV programmes in Vietnam have cost an estimated US$1,972 [1,447–2,747], US$2,344 [1,843–2,765], and US$248 [201–319] for each averted infection, death, and DALY, respectively.ConclusionsOur evaluation suggests that HIV programmes in Vietnam have most likely had benefits that are cost-effective. ART and direct HIV prevention were the most cost-effective interventions in reducing HIV disease burden.

Inflating Away the Public Debt? An Empirical Assessment

We propose and implement a method that provides quantitative estimates of the extent to which higher-than-expected inflation can lower the real value of outstanding government debt. Looking forward, we derive a formula for the debt burden that relies on detailed information about debt maturity and claimholders, and that uses option prices to construct risk-adjusted probability distributions for inflation at different horizons. The estimates suggest that it is unlikely that inflation will lower the US fiscal burden significantly, and that the effect of higher inflation is modest for plausible counterfactuals. If instead inflation is combined with financial repression that ex post extends the maturity of the debt, then the reduction in value can be significant.

Using a Virtual History Conference to Teach the Iraq War

In teaching the causes of the Iraq War, the use of “virtual history” can be employed in a conference setting in which different individuals are assigned to different plausible counterfactuals they use to construct virtual histories. The Iraq War lends itself to the virtual history approach because of the availability of many plausible counterfactuals at each of the various levels of analysis (individual, state/group, and international) at which the war has been explained. I report here on both the theoretical and pedagogical value of the virtual history conference in the study of the Iraq War.

Impact evaluation of school feeding programmes in Lao People's Democratic Republic

Despite the popularity and widespread implementation of school feeding programmes, evidence of their impact on school participation and nutritional status is mixed. In this study we evaluate feeding programmes in three districts of the Lao People's Democratic Republic. Feeding modalities included on-site feeding, take-home rations, and a combined modality. District-level implementation of the intervention sites and selective take-up presented considerable evaluation challenges. To address these, we use difference-in-difference estimators with propensity-score weighting to construct plausible counterfactuals. We find minimal evidence that school feeding increased enrolment or improved nutritional status. Several robustness checks and possible explanations for null findings are presented.

Propensity score matching and variations on the balancing test

Balancing tests are diagnostics designed for use with propensity score methods, a widely used non-experimental approach in the evaluation literature. Such tests provide useful information on whether plausible counterfactuals have been created. Currently, multiple balancing tests exist in the literature but it is unclear which is the most useful. This article highlights the poor size properties of commonly employed balancing tests and attempts to shed some light on the link between the results of balancing tests and bias of the evaluation estimator. The simulation results suggest that in scenarios where the conditional independence assumption holds, a permutation version of the balancing test described in Dehejia and Wahba (Rev Econ Stat 84:151–161, 2002) can be useful in applied study. The proposed test has good size properties. In addition, the test appears to have good power for detecting a misspecification in the link function and some power for detecting an omission of relevant non-linear terms involving variables that are included at a lower order.