Chapter 71 Econometric Evaluation of Social Programs, Part II: Using the Marginal Treatment Effect to Organize Alternative Econometric Estimators to Evaluate Social Programs, and to Forecast their Effects in New Environments

Monte Carlo simulation methods are increasingly being used to evaluate the property of statistical estimators in a variety of settings. The utility of these methods depends upon the existence of an appropriate data-generating process. Observational studies are increasingly being used to estimate the effects of exposures and interventions on outcomes. Conventional regression models allow for the estimation of conditional or adjusted estimates of treatment effects. There is an increasing interest in statistical methods for estimating marginal or average treatment effects. However, in many settings, conditional treatment effects can differ from marginal treatment effects. Therefore, existing data-generating processes for conditional treatment effects are of little use in assessing the performance of methods for estimating marginal treatment effects. In the current study, we describe and evaluate the performance of two different data-generation processes for generating data with a specified marginal odds ratio. The first process is based upon computing Taylor Series expansions of the probabilities of success for treated and untreated subjects. The expansions are then integrated over the distribution of the random variables to determine the marginal probabilities of success for treated and untreated subjects. The second process is based upon an iterative process of evaluating marginal odds ratios using Monte Carlo integration. The second method was found to be computationally simpler and to have superior performance compared to the first method.

This paper reviews four treatment parameters that have become commonly used in the program evaluation literature: the average treatment effect, the effect of treatment on the treated, the local average treatment effect, and the marginal treatment effect. We derive simply computed closed-form expressions for these treatment parameters in a latent variable framework with Gaussian error terms. These parameters can be estimated using nothing more than output from a standard two-step procedure. We also briefly describe recent work that seeks to go beyond mean effects and estimate the distributions associated with various outcome gains. The techniques presented in the paper are applied to estimate the return to some form of college education for various populations using data from the National Longitudinal Survey of Youth.

This paper reviews four treatment parameters that have become commonly used in the program evaluation literature: the average treatment effect, the effect of treatment on the treated, the local average treatment effect, and the marginal treatment effect. We derive simply computed closed‐form expressions for these treatment parameters in a latent variable framework with Gaussian error terms. These parameters can be estimated using nothing more than output from a standard two‐step procedure. We also briefly describe recent work that seeks to go beyond mean effects and estimate the distributions associated with various outcome gains. The techniques presented in the paper are applied to estimate the return to some form of college education for various populations using data from the National Longitudinal Survey of Youth.

Covariate adjustment in RCTs results in increased power to detect conditional effects compared with the power to detect unadjusted or marginal effects

This paper uses official Italian micro data and different methods to estimate, in the framework of potential outcomes, the marginal return to college education allowing for heterogeneous returns and for self-selection into higher education. Specifically, the paper is focused on the estimation of heterogeneity of average treatment effect (ATE) on a cohort of college and high school graduates using the 2008 survey on household, income and wealth of the Bank of Italy. Methodologically, this study was carried out by using both propensity-score-based (PS-based) methods and a new approach based on marginal treatment effects (MTE), recently proposed by Heckman and his associates as a useful strategy when the ignorability assumption may be violated. In the PS-based approach, heterogeneous treatment effects are estimated in three different manners: the traditional stratification approach (propensity score strata), the regression adjustment within propensity score strata and, finally, a non-parametric smoothing approach. In the MTE approach, the treatment effect heterogeneity across individuals is estimated in a parametric as well as a semi-parametric strategy. Our empirical analysis shows that the estimated heterogeneity is substantial: following MTE based results (quite representative of other methods) the return to college graduation for a randomly selected individual varies from as high as 20 % (for persons who would add one fifth of wage from graduating college) to as low as −22 % (for persons who would lose from college graduation), suggesting that returns are higher for individuals more likely to attend college. Furthermore, the results of different methods show very low (point) estimates of ATE: average college returns vary from 3.5 % by the PS-smoothing method to 1.8 % by the parametric MTE method, which also leads a greater treatment effect on treated (5.5 %), a moderate, but significant sorting gain and a negligible selection bias.

An essential feature common to all empirical social research is variability across units of analysis. Individuals differ not only in background characteristics, but also in how they respond to a particular treatment, intervention, or stimulation. Moreover, individuals may self-select into treatment on the basis of their anticipated treatment effects. To study heterogeneous treatment effects in the presence of self-selection, Heckman and Vytlacil (1999, 2001a, 2005, 2007b) have developed a structural approach that builds on the marginal treatment effect (MTE). In this paper, we extend the MTE-based approach through a redefinition of MTE. Specifically, we redefine MTE as the expected treatment effect conditional on the propensity score (instead of all observed covariates) as well as a latent variable representing unobserved resistance to treatment. The redefined MTE improves upon the original MTE in a number of aspects. First, while it is conditional on a unidimensional summary of covariates, it is sufficient to capture all of the treatment effect heterogeneity that is consequential for selection bias. Second, the new MTE is a bivariate function, and thus is easier to visualize than the original MTE. Third, as with the original MTE, the new MTE can also be used as a building block for evaluating standard causal estimands such as ATE and TT. However, the weights associated with the new MTE are simpler, more intuitive, and easier to compute. Finally, the redefined MTE immediately reveals treatment effect heterogeneity among individuals who are at the margin of treatment. As a result, it can be used to evaluate a wide range of policy changes with little analytical twist, and to design policy interventions that optimize the marginal benefits of treatment.

This paper studies measuring various average effects ofXonYin general structural systems with unobserved confoundersU, a potential instrumentZ, and a proxyWforU. We do not requireXorZto be exogenous given the covariates orWto be a perfect one‐to‐one mapping ofU. We study the identification of coefficients in linear structures as well as covariate‐conditioned average nonparametric discrete and marginal effects (e.g., average treatment effect on the treated), and local and marginal treatment effects. First, we characterize the bias, due to the omitted variablesU, of (nonparametric) regression and instrumental variables estimands, thereby generalizing the classic linear regression omitted variable bias formula. We then study the identification of the average effects ofXonYwhenUmay statistically depend onXandZ. These average effects are point identified if the average direct effect ofUonYis zero, in which case exogeneity holds, or ifWis a perfect proxy, in which case the ratio (contrast) of the average direct effect ofUonYto the average effect ofUonWis also identified. More generally, restricting how the average direct effect ofUonYcompares in magnitude and/or sign to the average effect ofUonWcan partially identify the average effects ofXonY. These restrictions on confounding are weaker than requiring benchmark assumptions, such as exogeneity or a perfect proxy, and enable a sensitivity analysis. After discussing estimation and inference, we apply this framework to study earnings equations.

An essential feature common to all empirical social research is variability across units of analysis. Individuals differ not only in background characteristics, but also in how they respond to a particular treatment, intervention, or stimulation. Moreover, individuals may self-select into treatment on the basis of their anticipated treatment effects. To study heterogeneous treatment effects in the presence of self-selection, Heckman and Vytlacil (1999, 2001a, 2005, 2007b) have developed a structural approach that builds on the marginal treatment effect (MTE). In this paper, we extend the MTE-based approach through a redefinition of MTE. Specifically, we redefine MTE as the expected treatment effect conditional on the propensity score (rather than all observed covariates) as well as a latent variable representing unobserved resistance to treatment. As with the original MTE, the new MTE can also be used as a building block for evaluating standard causal estimands. However, the weights associated with the new MTE are simpler, more intuitive, and easier to compute. Moreover, the new MTE is a bivariate function, and thus is easier to visualize than the original MTE. Finally, the redefined MTE immediately reveals treatment effect heterogeneity among individuals who are at the margin of treatment. As a result, it can be used to evaluate a wide range of policy changes with little analytical twist, and to design policy interventions that optimize the marginal benefits of treatment. We illustrate the proposed method by estimating heterogeneous economic returns to college with National Longitudinal Study of Youth 1979 (NLSY79) data.

This paper employs a heterogeneous education return framework to estimate the marginal treatment effect of college expansion on college premiums in China. Our findings are threefold. First, we capture heterogeneous college premiums after college expansion, especially in urban areas and provinces with abundant educational resources. Second, the treatment effects follow a quantitative sequence of ATT (Average Treatment Effect on Treated) > LATE (Local Average Treatment Effect) > ATE (Average Treatment Effect) > TNT (Average Treatment Effect on Non-Treated), which reconciles the puzzle that instrumental-variable estimates are always larger than OLS estimates in the literature. Third, we demonstrate that policies designed to increase the college entrance probability by a proportion will lead to groups with lower college premiums attending college, which is ineffective compared to those equalizing educational opportunities. This paper furthers our understanding of education premiums and sheds new light on the quality mechanism governing the impact of college expansion on college premiums.

Chapter 1 - Instrumental variables with unobserved heterogeneity in treatment effects

A number of statistical approaches have been proposed for incorporating supplemental information in randomized clinical trials. Existing methods often compare the marginal treatment effects to evaluate the degree of consistency between sources. Dissimilar marginal treatment effects would either lead to increased bias or down-weighting of the supplemental data. This represents a limitation in the presence of treatment effect heterogeneity, in which case the marginal treatment effect may differ between the sources solely due to differences between the study populations. We introduce the concept of covariate-adjusted exchangeability, in which differences in the marginal treatment effect can be explained by differences in the distributions of the effect modifiers. The potential outcomes framework is used to conceptualize covariate-adjusted and marginal exchangeability. We utilize a linear model and the existing multisource exchangeability models framework to facilitate borrowing when marginal treatment effects are dissimilar but covariate-adjusted exchangeability holds. We investigate the operating characteristics of our method using simulations. We also illustrate our method using data from two clinical trials of very low nicotine content cigarettes. Our method has the ability to incorporate supplemental information in a wider variety of situations than when only marginal exchangeability is considered.

This paper studies endogenous treatment effect models in which individuals are classified into unobserved groups based on heterogeneous treatment choice rules. Such heterogeneity may arise, for example, when multiple treatment eligibility criteria and different preference patterns exist. Using a finite mixture approach, we propose a marginal treatment effect (MTE) framework in which the treatment choice and outcome equations can be heterogeneous across groups. Under the availability of valid instrumental variables specific to each group, we show that the MTE for each group can be separately identified using the local instrumental variable method. Based on our identification result, we propose a two-step semiparametric procedure for estimating the group-wise MTE parameters. We first estimate the finite-mixture treatment choice model by a maximum likelihood method and then estimate the MTEs using a series approximation method. We prove that the proposed MTE estimator is consistent and asymptotically normally distributed. We illustrate the usefulness of the proposed method with an application to economic returns to college education.

This paper studies measuring various average effects of X on Y in general structural systems with unobserved confounders U, a potential instrument Z, and a proxy W for U. We do not require X or Z to be exogenous given the covariates or W to be a perfect one-to-one mapping of U. We study the identification of covariate-conditioned average random coefficients, average nonparametric discrete and marginal effects, average treatment effects for the population, treated, and untreated, and local and marginal treatment effects. First, we characterize the bias, due to the omitted variables U, of (nonparametric) regression and instrumental variables estimands, thereby generalizing the classic linear regression omitted variable bias formula. We then study the identification of the average effects of X on Y when U may statistically depend on X and Z. These average effects are point identified if the average direct effect of U on Y is zero, in which case exogeneity holds, or if W is a perfect proxy, in which case the ratio (contrast) of the average direct effect of U on Y to the average effect of U on W is also identified. More generally, restricting how the average direct effect of U on Y compares in magnitude and/or sign to the average effect of U on W can partially identify the average effects of X on Y. These restrictions on confounding are weaker than requiring benchmark assumptions, such as exogeneity or a perfect proxy, and enable a sensitivity analysis. After discussing estimation and inference, we apply this framework to study earning equations and production functions.

A computational approach to identification of treatment effects for policy evaluation

In this chapter, we review policy evaluation and Heckman and Vytlacil’s [2005, 2007a] (HV) strategy for linking marginal treatment effects to other average treatment effects including policy-relevant treatment effects. Recent innovations in the treatment effects literature including dynamic and general equilibrium considerations are mentioned briefly but in-depth study of these matters is not pursued. HV’s marginal treatment effects strategy is applied to the regulated report precision setting introduced in chapter 2, discussed in chapter 10, and continued in the next chapter. This analysis highlights the relative importance of probability distribution assignment to unobservables and quality of instruments.