Report of an Expert Panel to Review the Socio-Economic Models and Related Components Supporting the Development of Canada-Wide Standards (CWS) for Particulate Matter (PM) and Ozone To the Royal Society of Canada

Abstract

Canada-Wide Standards (CWS) for particulate matter (PM) and ozone may be the most ambitious environmental standards ever proposed in Canada. They have attracted considerable attention and debate. This report addresses the validity of the socioeconomic modeling aspects of the Canada-Wide Standards development process. Socioeconomic considerations are addressed in one of eight principles underlying the development and attainment of CWS, according to a CWS subagreement signed by the Canadian Council of Ministers of the Environment (CCME). Principle 3.1.7 states that measures to attain agreed-upon Canada-Wide Environmental Standards will be determined in a sustainable development context, recognizing environmental and socio-economic considerations. This work has been previously published by the Royal Society of Canada, Ottawa (Ontario), in June 2001, appearing as “Review of Socio-Economic Models and Related Components Supporting the Development of Canadian Standards for Particulate Matter and Ozone,” ISBN 0-920064-73-6. A CCME Framework for Socio-economic Analyses in Setting Environmental Standards (CCME, 1998) describes procedures and information requirements for socioeconomic assessments of potential or proposed environmental standards. This framework states that while it may not be possible or necessary to carry out all of the analytical steps because of time, data, or resource constraints, a partial assessment can produce information that is useful for policy deliberations. The framework also notes that socioeconomic findings are not intended to be prescriptive concerning decisions about environmental standards because other input factors such as toxicity, epidemiology, ecological consequences and geographical distribution of effects, and other equity considerations are also necessary and important to an informed choice with respect to standard setting. Socioeconomic considerations are also specified under Government of Canada regulatory policy requiring federal regulatory authorities to demonstrate that the benefits of regulatory requirements are greater than their costs. Regulatory authorities must “ensure that the benefits outweigh the costs to Canadians, their governments and businesses. In particular, when managing risks on behalf of Canadians, regulatory authorities must ensure that the limited resources available to government are used where they do the most good” (Government of Canada, 1999). This implies not only that benefits should be greater than costs, but also that benefits minus costs, or net benefits are to be maximized, which means an attempt should be made to make standards efficient. The objective of the expert panel process was to provide an independent, expert review and critique of the socioeconomic analyses (SEA)—in this case a cost-benefit analysis (CBA)—conducted in developing the Canada-Wide Standards on PM and ozone. Through a review of the models and associated data and assumptions used in the analyses, the panel was asked to produce a report to address the following questions: a. What are the strengths, merits, limitations, gaps, and the degree of uncertainties of the proposed approaches, models, and their inputs and outputs? b. By what means could the models and analytical approaches be improved, so as to minimize uncertainties and maximize the relevance, reliability, and utility of outputs? c. What other approaches and/or tools could be used to conduct these analyses? The benefits and costs associated with Canada-Wide Standards for PM and ozone are highly uncertain and controversial. Uncertainties are associated with each step in the analysis of benefits and costs—including the link between emission reductions and ambient air quality, the extent to which human health and the environment are affected by changes in ambient levels of pollutants, the economic values (as measures of preferences) associated with improvements in environmental and human health, accuracy of the emissions inventory and projections of what this inventory and other factors will be in a future baseline, and the scope and magnitude of economic costs associated with emission reductions, both to industry and to society. With uncertainties so pervasive, analysts are required to make many choices and assumptions in estimating costs and benefits. For example, while it is clear that the epidemiological association between PM and excess mortality is consistent and robust, there are many remaining gaps in current understanding of the relative toxicity of PM components and gaseous copollutants and the magnitude of potential life-shortening effects. These uncertainties introduce possible biases into the estimation of PM-related health benefits. The panel draws the following major conclusions from its review of the CBA undertaken for the development of CWS for PM and ozone and from the academic and policy literature relevant to this topic. In drawing these conclusions, the panel views the use of a structured approach to the examination of costs and benefits as a positive development in Canadian regulatory policy analysis. 1. The CWS socioeconomic analysis (SEA) was in fact limited to a cost-benefit analysis (CBA). Because the CWS implementation of the SEA process was judged by the panel to be limited to a CBA, it was reviewed as such. CBA is just one of many available decision support tools. The requirements of CBA in the CWS process depend on whether the purpose is to select an ambient air quality standard or to guide and evaluate the implementation process. The extent to which the results of the CBA can inform the CWS decision process is limited for various reasons including the following: • Provinces are required to establish implementation plans to ensure that CWS will be met. Therefore, when doing a prospective CBA, the measures to be implemented (i.e., revised emission control regulations) are necessarily undetermined, as are the compliance levels for the revised control regulations that will be achieved in various provinces. • The distributional impacts of both costs and benefits have not been assessed. The panel acknowledges that the CWS Development Committee for PM and Ozone describes the cost estimation as “preliminary and, in some instances, a cursory analysis used to provide a macro level order of magnitude perspective on the costs associated with the various optional levels for PM and ozone CWSs” and notes that caution should be exercised in their interpretation (Canada-Wide Standards Development Committee for PM and Ozone, 1999). Deficiencies in Canadian data and modeling capabilities and limited time and resources restricted the scope of the analysis that could be undertaken for the CWS process. 2. Credible CBA should be conducted to support the development of Canada-wide standards. The panel recommends that CBA be used to inform decision makers about the projected costs and anticipated benefits of CWS. CBA needs to be designed to distinguish between the costs and benefits of meeting alternative PM and ozone standards within the limits of current science. There are potential overlaps in the estimation of costs and benefits for PM and ozone because emission control strategies will impact both PM and ozone levels and it is not clear which components of the air pollution mix are responsible for the various health effects. These uncertainties in the CBA need to be clearly communicated. At its best, CBA provides the decision maker with a systematic identification, estimation, and measure of uncertainty of monetary values for the relevant costs and benefits of interest to decision makers and stakeholders. To be fully informative, the CBA results provided to stakeholders and decision makers need to adequately analyze and explain the major sources of uncertainty in the inputs of the CBA model projections and their likely effect on model outputs. 3. In view of the importance of the proposed regulatory decision, the CBA performed for the CWS for PM and ozone is deficient in relation to the state-of-the-art for CBA. If the CWS CBA was intended to provide an adequate basis for balancing costs and benefits and for influencing where the CWS should be set, this CBA was not up to the task. If the objective was strictly to confirm that costs were not exorbitant for CWS that were deemed to be both technically feasible and associated with some substantial benefits, then this CBA provided contributions toward those judgments. When judged against the elements of process and structure of CBA required for credibility as indicated in conclusions 4 and 5 that follows, the panel finds that the CWS CBA does not satisfy these requirements and does not meet a reasonable level of quality for a CBA to support a decision of this import. While the CWS CBA has some value as a scoping analysis and provides a limited degree of guidance for decision-makers, it requires substantial improvement to meet the criteria for credibility. 4. The process for using CBA in CWS needs improvement. A Discussion Paper on Particulate Matter (PM) and Ozone Canada-Wide Standard Scenarios for Consultation prepared by the Canada-Wide Standards Development Committee for PM and Ozone (May 1999) states that “in selecting PM and ozone CWS level scenarios for stakeholder considerations, an attempt was made to balance the anticipated benefits of improved air quality with the technological feasibility and costs of achieving those improvements.” But it is apparent that the standard CBA procedure of comparing incremental benefits for tighter standards with incremental costs was not done for the CWS CBA process. Timelines for the analysis were exceedingly short, and the CBA effort appeared to be underfunded, resulting in short-cuts that substantially reduced the credibility of the analysis. Reporting and communicating the CBA results was also ineffective, particularly in terms of conveying a clear understanding of what was done and why it was done as it was. 5. The panel has identified several CBA elements of primary concern that require attention in order to ensure the credibility of CWS CBA. The elements of primary concern for assessing credibility of CBA are the following: • Accuracy of emissions inventory data. • Accuracy of cost estimates. • Use of state-of-the-art air quality models. • Sufficiency of air quality monitoring. • Use of reasonable baseline assumptions for regulatory regime. • Inclusion of well-documented environmental conditions. • Inclusion of demographics in the CBA. • Adequate consideration of economic growth. • Selection of dose-response functions based on current weight of evidence. • Selection of valuation functions based on current weight of evidence. • Explicit expressions of uncertainty (measurement, model, and statistical). • Compatibility of scenarios with the form of standard (8-h and 24-h averaging times). • Inclusion of distributional analysis of costs and benefits (identification of sensitive subgroups, affected sectors). • Internal consistency of analyses (linking costs with benefits consistently). • Discussion of nonquantifiable endpoints. • Explanation for the choices of benefits and costs included. Performing CBA that meets these requirements will involve substantial investment on a continuing basis. Critics of the adequacy of the CWS CBA should support the generation of a knowledge base adequate to perform credible CBA. Generation of that knowledge base will require substantial investment of money, infrastructure and expertise. The panel has outlined its view on the limitations of the CBA undertaken for the development of CWS for PM and ozone. Given those concerns, the panel provides the following two conclusions on the measures of benefits and costs as calculated within the CWS CBA. 6. As in all CBA the estimates of benefits and costs are uncertain. Emerging analyses (in particular, of the tax interaction effect and the value of a statistical life) suggest that the costs associated with reducing emissions may be underestimated and the human health benefits overestimated. However, there are additional uncertainties that temper the impact of these emerging studies on the CBA. Most notably, the cost analysis performed in the CWS process is based on engineering estimates (resulting in overestimated costs) and the benefit measures do not include ecosystem effects (potentially large, but highly uncertain). The direction of the bias in net benefits depends on the weight placed on these factors. 7. The overall direction of the errors in benefits estimation is undetermined. The premature deaths reduced and broader range of health effects avoided by reducing PM ambient levels to the CWS PM standard are likely underestimated in the CWS analysis. However, the dollar value estimates for mortality reductions (based on value of statistical life, VSL) are very likely overestimated. The overall effect of these potential biases on the benefits realized from emission reductions is not clear from the current evidence. Based on these conclusions, the panel makes the following recommendations. 1. Capacity building. Given the identified deficiencies in the CWS CBA, the panel recommends the following: • Canada should build a capability for conducting CBA for CWS by improving emissions inventories, air quality modeling capability, air quality monitoring networks, socioeconomic modeling, human health data gathering, and developing economic analyses of health-environment interactions. This capacity building will require long-term financial support to build the infrastructure, as well as government and industry commitment to making these improvements. • Data and models should undergo continuing development and refinement with reporting and documentation at periodic intervals that are integrated within the timeframe for decision-making. This includes particularly the air quality valuation model (AQVM) used to estimate the health benefits of air quality improvements. As detailed later in this report, the AQVM may already be out of date in its choice of dose-response functions for estimating mortality risk reductions and for valuation of this health endpoint. The decision to update AQVM to include more recent work depends on the criteria of study inclusion, such as degree of peer review, number of confirmatory studies, etc. The panel recommends that such criteria used to develop the AQVM be reviewed and updated, as necessary. • Improve Canadian capacity for air quality modeling. Collaborations with other North American agencies and research groups should be encouraged and supported with long-term funding. • Inclusion of risk–risk trade-offs (estimation of damages associated with risks from pollutants that increase as a result of the pollutants of interest being reduced, e.g., UV-B radiation increasing as a result of ozone concentrations being decreased). • All CBA model specifications and input values (e.g., risk coefficients, health event valuations) used for the purposes of regulatory decision making should be fully transparent and readily accessible to all interested stakeholders and researchers. • An explicit procedural and consultative framework should be developed for CBA to inform the decision process. Informed decisions require dialogue and consultation between decision makers, stakeholders, and CBA analysts in an open, transparent process. The decision process should require consideration of results of CBA along with other inputs to the decision. • Funding should be allocated, and roles and responsibilities within the CWS process should be defined—including an external expert advisory body to review approaches, progress, etc. • Formal guidelines for considering evidence and making and communicating decisions should be developed. 2. Communication. Improved two-way communication concerning the assumptions, limitations, and uncertainties associated with the methods and results of CBA is needed between analysts and policymakers and between policymakers and the public. Clear communication of the conceptual underpinnings and limitations of valuation techniques and the interpretation of the results of cost and benefit studies is needed to correct prevailing misconceptions about the conduct and interpretation of these studies. 3. Cost-benefit analysis and other types of socioeconomic analyses. Socioeconomic analysis (SEA) includes a wide variety of social and economic analysis methods, of which cost-benefit analysis (CBA) is one example and is typically the foundation for other socioeconomic analyses. However, the panel recommends that CBA be conducted separately from broader socioeconomic analyses, including plant closures, unemployment, regional economic impacts, competitiveness, or inflation for broad-based rules. Such analyses often ignore labor and capital mobility and are not commensurate with values used in CBA and therefore, when provided alongside CBA estimates, the results of broader SEA models may give rise to double counting of benefits or costs. If effects are expected to be borne disproportionately by only a few sectors, these types of analyses are useful, but they should be presented as contributions to the assessment of the distribution of impacts. A more promising, but more resource-intensive, approach is the expansion of the CBA to a general equilibrium analysis to capture the costs of the tax interaction effect. (see Section 7.4) As for competitiveness analysis, shifts to imports may have positive environmental effects that would need to be taken into account, and in this sense, a broader SEA or general equilibrium framework would be useful. 4. Cost-benefit framework for analysis of environmental quality regulations. The panel endorses the use of a cost-benefit framework for the analysis of environmental regulation while recognizing the empirical limitations of CBA. The panel recommends: • Continued development of methods for accurate assessment of costs and benefits, including methods for the analysis of general equilibrium (including tax interaction) effects and international trade impacts of regulatory change. • Continued development of and communication regarding alternative decision-making frameworks, including multi-attribute methods, to be used as methods to “triangulate” with traditional CBA. • Investments in human capital in the area of CBA of environmental regulation so that policy makers and the Canadian public can be confident that cost and benefit measures accurately reflect Canadian values and public preferences as well as Canadian institutional arrangements.