Earth System Model Evaluation Tool Research Articles

Evaluation of native Earth system model output with ESMValTool v2.6.0

Abstract. Earth system models (ESMs) are state-of-the-art climate models that allow numerical simulations of the past, present-day, and future climate. To extend our understanding of the Earth system and improve climate change projections, the complexity of ESMs heavily increased over the last decades. As a consequence, the amount and volume of data provided by ESMs has increased considerably. Innovative tools for a comprehensive model evaluation and analysis are required to assess the performance of these increasingly complex ESMs against observations or reanalyses. One of these tools is the Earth System Model Evaluation Tool (ESMValTool), a community diagnostic and performance metrics tool for the evaluation of ESMs. Input data for ESMValTool needs to be formatted according to the CMOR (Climate Model Output Rewriter) standard, a process that is usually referred to as “CMORization”. While this is a quasi-standard for large model intercomparison projects like the Coupled Model Intercomparison Project (CMIP), this complicates the application of ESMValTool to non-CMOR-compliant climate model output. In this paper, we describe an extension of ESMValTool introduced in v2.6.0 that allows seamless reading and processing of “native” climate model output, i.e., operational output produced by running the climate model through the standard workflow of the corresponding modeling institute. This is achieved by an extension of ESMValTool's preprocessing pipeline that performs a CMOR-like reformatting of the native model output during runtime. Thus, the rich collection of diagnostics provided by ESMValTool is now fully available for these models. For models that use unstructured grids, a further preprocessing step required to apply many common diagnostics is regridding to a regular latitude–longitude grid. Extensions to ESMValTool's regridding functions described here allow for more flexible interpolation schemes that can be used on unstructured grids. Currently, ESMValTool supports nearest-neighbor, bilinear, and first-order conservative regridding from unstructured grids to regular grids. Example applications of this new native model support are the evaluation of new model setups against predecessor versions, assessing of the performance of different simulations against observations, CMORization of native model data for contributions to model intercomparison projects, and monitoring of running climate model simulations. For the latter, new general-purpose diagnostics have been added to ESMValTool that are able to plot a wide range of variable types. Currently, five climate models are supported: CESM2 (experimental; at the moment, only surface variables are available), EC-Earth3, EMAC, ICON, and IPSL-CM6. As the framework for the CMOR-like reformatting of native model output described here is implemented in a general way, support for other climate models can be easily added.

The E3SM Diagnostics Package (E3SM Diags v2.7): a Python-based diagnostics package for Earth system model evaluation

Abstract. The E3SM Diagnostics Package (E3SM Diags) is a modern, Python-based Earth system model (ESM) evaluation tool (with Python module name e3sm_diags), developed to support the Department of Energy (DOE) Energy Exascale Earth System Model (E3SM). E3SM Diags provides a wide suite of tools for evaluating native E3SM output, as well as ESM data on regular latitude–longitude grids, including output from Coupled Model Intercomparison Project (CMIP) class models. E3SM Diags is modeled after the National Center for Atmospheric Research (NCAR) Atmosphere Model Working Group (AMWG, 2022) diagnostics package. In its version 1 release, E3SM Diags included a set of core essential diagnostics to evaluate the mean physical climate from model simulations. As of version 2.7, more process-oriented and phenomenon-based evaluation diagnostics have been implemented, such as analysis of the quasi-biennial oscillation (QBO), the El Niño–Southern Oscillation (ENSO), streamflow, the diurnal cycle of precipitation, tropical cyclones, ozone and aerosol properties. An in situ dataset from DOE's Atmospheric Radiation Measurement (ARM) program has been integrated into the package for evaluating the representation of simulated cloud and precipitation processes. This tool is designed with enough flexibility to allow for the addition of new observational datasets and new diagnostic algorithms. Additional features include customizable figures; streamlined installation, configuration and execution; and multiprocessing for fast computation. The package uses an up-to-date observational data repository maintained by its developers, where recent datasets are added to the repository as they become available. Finally, several applications for the E3SM Diags module were introduced to fit a diverse set of use cases from the scientific community.

ERA5 and ERA-Interim Data Processing for the GlobWat Global Hydrological Model

The reproducibility of computational hydrology is gaining attention among hydrologists. Reproducibility requires open and reusable code and data, allowing users to verify results and process new datasets. The creation of input files for global hydrological models (GHMs) requires complex high-resolution gridded dataset processing, limiting the model’s reproducibility to groups with advanced programming skills. GlobWat is one of these GHMs, which was developed by the Food and Agriculture Organization (FAO) to assess irrigation water use. Although the GlobWat code and sample input data are available, the methods for pre-processing model inputs are not available. Here, we present a set of open-source Python and YAML scripts within the Earth System Model Evaluation Tool (ESMValTool) that provide a formalized technique for developing and processing GlobWat model weather inputs. We demonstrate the use of these scripts with the ERA5 and ERA-Interim datasets from the European Centre for Medium-Range Weather Forecasts (ECMWF). To demonstrate the advantage of using these scripts, we ran the GlobWat model for 30 years for the entire world. The focus of the evaluation was on the Urmia Lake Basin in Iran. The validation of the model against the observed discharge in this basin showed that the combination of ERA5 and the De Bruin reference evaporation method yields the best GlobWat performance. Moreover, the scripts allowed us to examine the causes behind the differences in model outcomes.

Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for extreme events, regional and impact evaluation, and analysis of Earth system models in CMIP

Abstract. This paper complements a series of now four publications that document the release of the Earth System Model Evaluation Tool (ESMValTool) v2.0. It describes new diagnostics on the hydrological cycle, extreme events, impact assessment, regional evaluations, and ensemble member selection. The diagnostics are developed by a large community of scientists aiming to facilitate the evaluation and comparison of Earth system models (ESMs) which are participating in the Coupled Model Intercomparison Project (CMIP). The second release of this tool aims to support the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). Furthermore, datasets from other models and observations can be analysed. The diagnostics for the hydrological cycle include several precipitation and drought indices, as well as hydroclimatic intensity and indices from the Expert Team on Climate Change Detection and Indices (ETCCDI). The latter are also used for identification of extreme events, for impact assessment, and to project and characterize the risks and impacts of climate change for natural and socio-economic systems. Further impact assessment diagnostics are included to compute daily temperature ranges and capacity factors for wind and solar energy generation. Regional scales can be analysed with new diagnostics implemented for selected regions and stochastic downscaling. ESMValTool v2.0 also includes diagnostics to analyse large multi-model ensembles including grouping and selecting ensemble members by user-specified criteria. Here, we present examples for their capabilities based on the well-established CMIP Phase 5 (CMIP5) dataset.

Spatially resolved evaluation of Earth system models with satellite column-averaged CO<sub>2</sub>

Abstract. Earth system models (ESMs) participating in the Coupled Model Intercomparison Project Phase 5 (CMIP5) showed large uncertainties in simulating atmospheric CO2 concentrations. We utilize the Earth System Model Evaluation Tool (ESMValTool) to evaluate emission-driven CMIP5 and CMIP6 simulations with satellite data of column-average CO2 mole fractions (XCO2). XCO2 time series show a large spread among the model ensembles both in CMIP5 and CMIP6. Compared to the satellite observations, the models have a bias of +25 to −20 ppmv in CMIP5 and +20 to −15 ppmv in CMIP6, with the multi-model mean biases at +10 and +2 ppmv, respectively. The derived mean atmospheric XCO2 growth rate (GR) of 2.0 ppmv yr−1 is overestimated by 0.4 ppmv yr−1 in CMIP5 and 0.3 ppmv yr−1 in CMIP6 for the multi-model mean, with a good reproduction of the interannual variability. All models capture the expected increase of the seasonal cycle amplitude (SCA) with increasing latitude, but most models underestimate the SCA. Any SCA derived from data with missing values can only be considered an “effective” SCA, as the missing values could occur at the peaks or troughs. The satellite data are a combined data product covering the period 2003–2014 based on the Scanning Imaging Absorption Spectrometer for Atmospheric Chartography (SCIAMACHY)/Envisat (2003–2012) and Thermal And Near infrared Sensor for carbon Observation Fourier transform spectrometer/Greenhouse Gases Observing Satellite (TANSO-FTS/GOSAT) (2009–2014) instruments. While the combined satellite product shows a strong negative trend of decreasing effective SCA with increasing XCO2 in the northern midlatitudes, both CMIP ensembles instead show a non-significant positive trend in the multi-model mean. The negative trend is reproduced by the models when sampling them as the observations, attributing it to sampling characteristics. Applying a mask of the mean data coverage of each satellite to the models, the effective SCA is higher for the SCIAMACHY/Envisat mask than when using the TANSO-FTS/GOSAT mask. This induces an artificial negative trend when using observational sampling over the full period, as SCIAMACHY/Envisat covers the early period until 2012, with TANSO-FTS/GOSAT measurements starting in 2009. Overall, the CMIP6 ensemble shows better agreement with the satellite data than the CMIP5 ensemble in all considered quantities (XCO2, GR, SCA and trend in SCA). This study shows that the availability of column-integral CO2 from satellite provides a promising new way to evaluate the performance of Earth system models on a global scale, complementing existing studies that are based on in situ measurements from single ground-based stations.

Quantifying Progress Across Different CMIP Phases With the ESMValTool

AbstractMore than 40 model groups worldwide are participating in the Coupled Model Intercomparison Project Phase 6 (CMIP6), providing a new and rich source of information to better understand past, present, and future climate change. Here, we use the Earth System Model Evaluation Tool (ESMValTool) to assess the performance of the CMIP6 ensemble compared to the previous generations CMIP3 and CMIP5. While CMIP5 models did not capture the observed pause in the increase in global mean surface temperature between 1998 and 2013, the historical CMIP6 simulations agree well with the observed recent temperature increase, but some models have difficulties in reproducing the observed global mean surface temperature record of the second half of the twentieth century. While systematic biases in annual mean surface temperature and precipitation remain in the CMIP6 multimodel mean, individual models and high‐resolution versions of the models show significant reductions in many long‐standing biases. Some improvements are also found in the vertical temperature, water vapor, and zonal wind speed distributions, and root‐mean‐square errors for selected fields are generally smaller with reduced intermodel spread and higher average skill in the correlation patterns relative to observations. An emerging property of the CMIP6 ensemble is a higher effective climate sensitivity with an increased range between 2.3 and 5.6 K. A possible reason for this increase in some models is improvements in cloud representation resulting in stronger shortwave cloud feedbacks than in their predecessor versions.

Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for emergent constraints and future projections from Earth system models in CMIP

Abstract. The Earth System Model Evaluation Tool (ESMValTool), a community diagnostics and performance metrics tool for evaluation and analysis of Earth system models (ESMs), is designed to facilitate a more comprehensive and rapid comparison of single or multiple models participating in the Coupled Model Intercomparison Project (CMIP). The ESM results can be compared against observations or reanalysis data as well as against other models including predecessor versions of the same model. The updated and extended version (v2.0) of the ESMValTool includes several new analysis scripts such as large-scale diagnostics for evaluation of ESMs as well as diagnostics for extreme events, regional model and impact evaluation. In this paper, the newly implemented climate metrics such as effective climate sensitivity (ECS) and transient climate response (TCR) as well as emergent constraints for various climate-relevant feedbacks and diagnostics for future projections from ESMs are described and illustrated with examples using results from the well-established model ensemble CMIP5. The emergent constraints implemented include constraints on ECS, snow-albedo effect, climate–carbon cycle feedback, hydrologic cycle intensification, future Indian summer monsoon precipitation and year of disappearance of summer Arctic sea ice. The diagnostics included in ESMValTool v2.0 to analyze future climate projections from ESMs further include analysis scripts to reproduce selected figures of chapter 12 of the Intergovernmental Panel on Climate Change's (IPCC) Fifth Assessment Report (AR5) and various multi-model statistics.

Earth System Model Evaluation Tool (ESMValTool) v2.0 – an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

Abstract. The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of (1) an easy-to-install, well-documented Python package providing the core functionalities (ESMValCore) that performs common preprocessing operations and (2) a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability, the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top–down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klimarechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5.

Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview

Abstract. This paper describes the second major release of the Earth System Model Evaluation Tool (ESMValTool), a community diagnostic and performance metrics tool for the evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). Compared to version 1.0, released in 2016, ESMValTool version 2.0 (v2.0) features a brand new design, with an improved interface and a revised preprocessor. It also features a significantly enhanced diagnostic part that is described in three companion papers. The new version of ESMValTool has been specifically developed to target the increased data volume of CMIP Phase 6 (CMIP6) and the related challenges posed by the analysis and the evaluation of output from multiple high-resolution or complex ESMs. The new version takes advantage of state-of-the-art computational libraries and methods to deploy an efficient and user-friendly data processing. Common operations on the input data (such as regridding or computation of multi-model statistics) are centralized in a highly optimized preprocessor, which allows applying a series of preprocessing functions before diagnostics scripts are applied for in-depth scientific analysis of the model output. Performance tests conducted on a set of standard diagnostics show that the new version is faster than its predecessor by about a factor of 3. The performance can be further improved, up to a factor of more than 30, when the newly introduced task-based parallelization options are used, which enable the efficient exploitation of much larger computing infrastructures. ESMValTool v2.0 also includes a revised and simplified installation procedure, the setting of user-configurable options based on modern language formats, and high code quality standards following the best practices for software development.

TheDiaTo (v1.0) – a new diagnostic tool for water, energy and entropy budgets in climate models

Abstract. This work presents the Thermodynamic Diagnostic Tool (TheDiaTo), a novel diagnostic tool for investigating the thermodynamics of climate systems with a wide range of applications, from sensitivity studies to model tuning. It includes a number of modules for assessing the internal energy budget, the hydrological cycle, the Lorenz energy cycle and the material entropy production. The routine takes as inputs energy fluxes at the surface and at the top of the atmosphere (TOA), which allows for the computation of energy budgets at the TOA, the surface and in the atmosphere as a residual. Meridional enthalpy transports are also computed from the divergence of the zonal mean energy budget from which the location and intensity of the maxima in each hemisphere are calculated. Rainfall, snowfall and latent heat fluxes are received as inputs for computation of the water mass and latent energy budgets. If a land–sea mask is provided, the required quantities are separately computed over continents and oceans. The diagnostic tool also computes the annual Lorenz energy cycle (LEC) and its storage and conversion terms by hemisphere and as a global mean. This is computed from three-dimensional daily fields of horizontal wind velocity and temperature in the troposphere. Two methods have been implemented for the computation of the material entropy production: one relying on the convergence of radiative heat fluxes in the atmosphere (indirect method) and the other combining the irreversible processes occurring in the climate system, particularly heat fluxes in the boundary layer, the hydrological cycle and the kinetic energy dissipation as retrieved from the residuals of the LEC (direct method). A version of these diagnostics has been developed as part of the Earth System Model eValuation Tool (ESMValTool) v2.0a1 in order to assess the performances of CMIP6 model simulations, and it will be available in the next release. The aim of this software is to provide a comprehensive picture of the thermodynamics of the climate system, as reproduced in the state-of-the-art coupled general circulation models. This can prove useful for better understanding anthropogenic and natural climate change, paleoclimatic climate variability, and climatic tipping points.

Process-level improvements in CMIP5 models and their impact on tropical variability, the Southern Ocean, and monsoons

Abstract. The performance of updated versions of the four earth system models (ESMs) CNRM, EC-Earth, HadGEM, and MPI-ESM is assessed in comparison to their predecessor versions used in Phase 5 of the Coupled Model Intercomparison Project. The Earth System Model Evaluation Tool (ESMValTool) is applied to evaluate selected climate phenomena in the models against observations. This is the first systematic application of the ESMValTool to assess and document the progress made during an extensive model development and improvement project. This study focuses on the South Asian monsoon (SAM) and the West African monsoon (WAM), the coupled equatorial climate, and Southern Ocean clouds and radiation, which are known to exhibit systematic biases in present-day ESMs. The analysis shows that the tropical precipitation in three out of four models is clearly improved. Two of three updated coupled models show an improved representation of tropical sea surface temperatures with one coupled model not exhibiting a double Intertropical Convergence Zone (ITCZ). Simulated cloud amounts and cloud–radiation interactions are improved over the Southern Ocean. Improvements are also seen in the simulation of the SAM and WAM, although systematic biases remain in regional details and the timing of monsoon rainfall. Analysis of simulations with EC-Earth at different horizontal resolutions from T159 up to T1279 shows that the synoptic-scale variability in precipitation over the SAM and WAM regions improves with higher model resolution. The results suggest that the reasonably good agreement of modeled and observed mean WAM and SAM rainfall in lower-resolution models may be a result of unrealistic intensity distributions.

Benchmarking CMIP5 models with a subset of ESA CCI Phase 2 data using the ESMValTool

The Coupled Model Intercomparison Project (CMIP) is now moving into its sixth phase and aims at a more routine evaluation of the models as soon as the model output is published to the Earth System Grid Federation (ESGF). To meet this goal the Earth System Model Evaluation Tool (ESMValTool), a community diagnostics and performance metrics tool for the systematic evaluation of Earth system models (ESMs) in CMIP, has been developed and a first version (1.0) released as open source software in 2015. Here, an enhanced version of the ESMValTool is presented that exploits a subset of Essential Climate Variables (ECVs) from the European Space Agency's Climate Change Initiative (ESA CCI) Phase 2 and this version is used to demonstrate the value of the data for model evaluation. This subset includes consistent, long-term time series of ECVs obtained from harmonized, reprocessed products from different satellite instruments for sea surface temperature, sea ice, cloud, soil moisture, land cover, aerosol, ozone, and greenhouse gases. The ESA CCI data allow extending the calculation of performance metrics as summary statistics for some variables and add an important alternative data set in other cases where observations are already available. The provision of uncertainty estimates on a per grid basis for the ESA CCI data sets is used in a new extended version of the Taylor diagram and provides important additional information for a more objective evaluation of the models. In our analysis we place a specific focus on the comparability of model and satellite data both in time and space. The ESA CCI data are well suited for an evaluation of results from global climate models across ESM compartments as well as an analysis of long-term trends, variability and change in the context of a changing climate. The enhanced version of the ESMValTool is released as open source software and ready to support routine model evaluation in CMIP6 and at individual modeling centers.

ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP

Abstract. A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.