What Can Meta‐Analyses Tell Us About the Reliability of Life Cycle Assessment for Decision Support?

Abstract

The body of life cycle assessment (LCA) literature is vast and has grown over the last decade at a dauntingly rapid rate. Many LCAs have been published on the same or very similar technologies or products, in some cases leading to hundreds of publications. One result is the impression among decision makers that LCAs are inconclusive, owing to perceived and real variability in published estimates of life cycle impacts. Despite the extensive available literature and policy need for more conclusive assessments, only modest attempts have been made to synthesize previous research. A significant challenge to doing so are differences in characteristics of the considered technologies and inconsistencies in methodological choices (e.g., system boundaries, coproduct allocation, and impact assessment methods) among the studies that hamper easy comparisons and related decision support. An emerging trend is meta-analysis of a set of results from LCAs, which has the potential to clarify the impacts of a particular technology, process, product, or material and produce more robust and policy-relevant results. Meta-analysis in this context is defined here as an analysis of a set of published LCA results to estimate a single or multiple impacts for a single technology or a technology category, either in a statistical sense (e.g., following the practice in the biomedical sciences) or by quantitative adjustment of the underlying studies to make them more methodologically consistent. One example of the latter approach was published in Science by Farrell and colleagues (2006) clarifying the net energy and greenhouse gas (GHG) emissions of ethanol, in which adjustments included the addition of coproduct credit, the addition and subtraction of processes within the system boundary, and a reconciliation of differences in the definition of net energy metrics. Such adjustments therefore provide an even playing field on which all studies can be considered and at the same time specify the conditions of the playing field itself. Understanding the conditions under which a meta-analysis was conducted is important for proper interpretation of both the magnitude and variability in results. This special supplemental issue of the Journal of Industrial Ecology includes 12 high-quality meta-analyses and critical reviews of LCAs that advance understanding of the life cycle environmental impacts of different technologies, processes, products, and materials. Also published are three contributions on methodology and related discussions of the role of meta-analysis in LCA. The goal of this special supplemental issue is to contribute to the state of the science in LCA beyond the core practice of producing independent studies on specific products or technologies by highlighting the ability of meta-analysis of LCAs to advance understanding in areas of extensive existing literature. The inspiration for the issue came from a series of meta-analyses of life cycle GHG emissions from electricity generation technologies based on research from the LCA Harmonization Project1 of the National Renewable Energy Laboratory (NREL), a laboratory of the U.S. Department of Energy, which also provided financial support for this special supplemental issue. (See the editorial from this special supplemental issue [Lifset 2012], which introduces this supplemental issue and discusses the origins, funding, peer review, and other aspects.) The first article on reporting considerations for meta-analyses/critical reviews for LCA is from Heath and Mann (2012), who describe the methods used and experience gained in NREL's LCA Harmonization Project, which produced six of the studies in this special supplemental issue. Their harmonization approach adapts key features of systematic review to identify and screen published LCAs followed by a meta-analytical procedure to adjust published estimates to ones based on a consistent set of methods and assumptions to allow interstudy comparisons and conclusions to be made. In a second study on methods, Zumsteg and colleagues (2012) propose a checklist for a standardized technique to assist in conducting and reporting systematic reviews of LCAs, including meta-analysis, that is based on a framework used in evidence-based medicine. Widespread use of such a checklist would facilitate planning successful reviews, improve the ability to identify systematic reviews in literature searches, ease the ability to update content in future reviews, and allow more transparency of methods to ease peer review and more appropriately generalize findings. Finally, Zamagni and colleagues (2012) propose an approach, inspired by a meta-analysis, for categorizing main methodological topics, reconciling diverging methodological developments, and identifying future research directions in LCA. Their procedure involves the carrying out of a literature review on articles selected according to predefined criteria. The analysis highlights the need for improvement in LCA practicability and model fidelity. The 12 meta-analyses in this special supplemental issue are listed in Table 1. These studies elucidate the GHG emissions of alternative electricity generation technologies (coal, photovoltaics [PVs; crystalline silicon and thin-film PVs in two articles], concentrating solar power [CSP], wind, and nuclear) and carbon-capture and storage, as well as LCA applications to biobased materials and computers. Each study began with the identification of relevant LCAs and followed with an analysis of a subset selected on the basis of screening criteria described in each manuscript. As shown in Table 1, this subset ranged from 5 to 53 LCAs; however, in most cases the meta-analyses covered a larger number of technology systems, as each individual LCA in the selected set quite often compared multiple systems. Most of the 12 meta-analyses focus on the life cycle GHG emissions of electricity generation by different sources. Coal-fired and nuclear electricity generation systems are harmonized by Whitaker and colleagues (2012) and Warner and Heath (2012). Harmonization of 53 utility-scale coal-fired electricity generation LCAs by Whitaker and colleagues (2012) finds that approximately 99% of life cycle GHG emissions are directly related to the coal fuel cycle (including combustion) such that a first-order estimate of life cycle GHG emissions could be based on knowledge of the technology type, coal mine emissions, thermal efficiency, and the combustion carbon dioxide emission factor alone without requiring full LCAs. This is in contrast to the findings of Warner and Heath (2012) for light water nuclear power. Significant variability remained after harmonization by system boundary and performance parameters, which could be qualitatively explained by variations in assumed primary source energy mix, uranium ore grade, and the selected LCA method (i.e., process chain vs. economic input-output LCA methods). Solar electricity generation systems are explored in three articles. First, Burkhardt and colleagues (2012) harmonize ten CSP system LCAs and illustrate the use of a two-level harmonization process for parabolic trough and power tower technologies. Utilizing so-called light harmonization, when the solar fraction and several other performance parameters of both technologies are harmonized, a significant reduction in variability compared to the published estimates of life cycle GHG emissions is revealed. A more intensive level of harmonization was then employed on a smaller pool of studies, which included application of consistent global warming intensities of materials in the life cycle inventory and inclusion of auxiliary natural gas and electricity consumption, revealing an even greater reduction in the estimated variability but an increased central tendency compared to the lightly harmonized results (owing to the inclusion of required auxiliary natural gas and electricity consumption, which are often incorrectly excluded from CSP LCAs). In a second investigation of solar electricity by Kim and colleagues (2012), five studies are used to harmonize amorphous silicon (a-Si), cadmium telluride (CdTe), and copper indium gallium diselenide (CIGS) photovoltaic systems by adjusting efficiency, irradiation, performance ratio, balance of system, and lifetime. Although the adjustment of all of these parameters is found to contribute to a reduced estimate of variability, the importance of irradiation, efficiency, and lifetime are highlighted. Similarly, Hsu and colleagues (2012) harmonize 13 LCAs on crystalline silicon photovoltaic electricity generation by adjusting efficiency, irradiation, the performance ratio, and lifetime and also identify irradiation and lifetime as drivers of variability in the results. The final three studies on electricity generation investigate wind power systems. Price and Kendall (2012) provide a systematic review of LCAs to investigate life cycle GHG emissions for modern wind turbines in a wide study region (studies are from Australia, Brazil, Canada, Europe, India, New Zealand, Taiwan, and the United States). Adjustments made in turbine size, geographic location, and end-of-life treatment in addition to those intended to produce a consistent system boundary across studies are critiqued. The results of the critique are combined with a requirement that LCAs chosen for review include only those with original LCA data. The 18 LCAs passing the screening criteria are then assessed in a scoring rubric designed to assist in an understanding of consistency in LCA meta-analyses. Next, Padey and colleagues (2012) provide a meta-analysis of life cycle GHG emissions of wind electricity on the basis of 19 LCAs for systems recently manufactured and operated in Europe. These authors use a screening approach somewhat similar to that of Price and colleagues, use adjustment levels representing Europe, and find manufacturing materials, load factor as a function of wind speed, and product lifetime to be most influential. Finally, estimates of life cycle GHG emissions from wind electricity are harmonized by Dolan and Heath (2012) in an analysis of 49 LCAs from a global study region. These authors employ a light harmonization approach with less of a focus on differences in manufacturing and end-of-life management. Adjustment of the capacity factor (i.e., the load factor), operating lifetime, and system boundaries revealed that harmonization by capacity factor resulted in the largest reduction in variability in life cycle GHG emissions. Note also that the wind meta-analyses of, for example, Padey and colleagues and Dolan and Heath must not be compared on the basis of the resulting means, as each harmonize/adjust parameters to a different set of conditions. Schreiber and colleagues (2012) performed a meta-analysis of 15 LCA studies on electricity generation with three carbon capture and storage technologies (postcombustion, oxyfuel, and precombustion) with a focus on GHG reduction for different regions, fuels, and time horizons. They present a condensed overview of methodological variations, findings, and conclusions gathered from the 15 LCAs. Considering all capture technologies, time horizons, or fuels evaluated, the potential climate benefits of these technologies are counterbalanced by impacts on a range of other environmental categories (e.g., acidification, eutrophication, and photochemical ozone creation). The results are significantly sensitive to three parameter sets: power plant efficiency and energy penalty of the capture process, carbon dioxide capture efficiency and purity, and fuel origin and composition. For the studies beyond those related to electricity generation, Weiss and colleagues (2012) perform a comprehensive meta-analysis on the environmental benefits and burdens of biobased materials, in which 44 LCAs were reviewed. The authors found that biobased materials save both energy and GHG emissions relative to their fossil counterparts. Conversely, biobased materials may increase eutrophication and stratospheric ozone depletion. Differences in impacts on acidification and photochemical ozone formation are inconclusive. The large uncertainty of individual LCA studies highlights the difficulties in drawing general conclusions about the relative environmental merits between different materials. Teehan and colleagues (2012) provide a systematic review of LCAs on desktop computers aimed at understanding variability and discrepancies among published studies. Specifically, whereas the majority of studies find that the use phase dominates GHG emissions, three studies disagree with the majority. Given this, Teehan and colleagues select and decompose 13 LCAs to the system component, life cycle phase, and inventory flow levels. Their decomposition to the component level was hampered by a lack of transparency in the published studies that did not allow assessment or adjustment of the underlying parameters. Using published data, they find the manufacturing phase at a smaller but substantial level of contribution to the overall results. Alternatively, Teehan and colleagues found much higher transparency in the use-phase data within the studies reviewed. They reveal that assumptions concerning the hours of daily use directly correlate with the dominance of the use phase, and they question the general applicability of low use estimates (e.g., within the context of plug load measurements). As a result, they identify the use phase as dominant for energy demand and contribution to climate change, with the only exception being regions with low GHG electricity generation. Finally, Gambeta and colleagues (2012) critically review 12 LCAs on consumer imaging equipment using an International Organization for Standardization (ISO) 14040 framework to identify common practices, limitations, and opportunities for improvement and standardization. Their analysis suggests that comparisons across studies are significantly hampered by variability in methods and reporting. They conclude that standardization of the functional unit and the assumptions that are interwoven with it has a high potential to increase quantitative comparability across studies. This special supplemental issue makes clear that meta-analysis is very useful in clarifying an understanding of impact magnitude and variability and of the underlying technological parameters that drive the results. Meta-analyses of LCAs are becoming more widely recognized in the field for these virtues through special sessions at conferences, for instance the 2010 International Life Cycle Assessment (InLCA) conference (Heath et al. 2010) and the upcoming 2012 Society of Environmental Toxicology and Chemistry (SETAC) World Congress (Heath and Brandão 2012), the call for papers for this special issue that produced many excellent submissions, including some not yet published, and the few publications that preceded the aforementioned (e.g., the multiregression analysis of Lenzen and Munksgaard 2002). However, the results of LCA studies—and the subsequent decisions they support—are dependent on a wide range of factors that make each LCA study unique. This may limit the use of LCA for decision support, unless LCA studies abide by the same methodological guidelines and principles and are thus consistent and comparable. Data quality is often cited as the major bottleneck of robust LCAs, but other factors play a large role. The variability of LCA results does not depend solely on the variability of the data employed, but rather on a range of factors. Methodological choices related to scope, system boundaries, allocation, choice of impact assessment method, as well as other assumptions, make LCA a tool that often generates uncertain outcomes. All these factors decrease the impact LCA could have in supporting decisions in both public policy and business domains. Therefore, more harmonization needs to take place. More standardization could include the adoption of a clear set of criteria to facilitate analysis of how data quality, scope, assumptions, key findings, and the like affect the results, and for the complete reporting of all key assumptions and methods. Therefore the robustness of LCA studies cannot be assessed without an uncertainty analysis. On a global level, the ISO 14040–44 series (ISO 2006a, 2006b) attempt to provide some level of standardization and harmonization in both methodological and procedural choices and reporting. Recent developments that complement and go beyond the ISO standards come from the European Commission's Joint Research Centre, including the International Reference Life Cycle Data System (ILCD) handbook (European Commission 2010) and an LCA directory2 containing several LCA studies and using clear fields that structure and facilitate analysis in terms of suitability for consideration in a policy-support context or for meta-analysis purposes. Additional developments include those under the United Nations Environment Programme (UNEP)-SETAC Life Cycle Initiative.3 Many journals have published numerous LCA studies in recent years. In order to ensure the quality and relevance of LCA studies, not only is peer review an important step, but so is conformity to a common set of rules for performing an LCA. Even though there is still no commonly accepted and applied global standard, not even ISO, the articles in this special supplemental issue show that LCA results are, more often than not, pointing in the same direction. This suggests that LCA is already relevant for supporting decisions, though it could be strengthened through meta-analysis of previous research and methodological guidelines for the conduct of future LCAs. Support for this special supplemental issue was provided by the U.S. Department of Energy through the National Renewable Energy Laboratory. Miguel Brandão was a scientific officer at the Joint Research Centre of the European Commission in Ispra, Italy, when this issue was prepared. He is currently working at the International Life Cycle Academy, Barcelona, Spain. He is also an associate editor for LCA of the Journal of Industrial Ecology, and a member of the Steering Committee of SETAC Europe LCA. Garvin Heath is a senior scientist and member of the Technology Systems and Sustainability Analysis Group in the Strategic Energy Analysis Center of the U.S. Department of Energy's National Renewable Energy Laboratory (NREL), Golden, Colorado, USA. He led the team that conducted NREL's LCA Harmonization Project. Joyce Cooper is an associate professor of mechanical engineering at the University of Washington, Seattle, Washington, USA.