Performance Of Individual Models Research Articles

How accurate are estimates of glacier ice thickness? Results from ITMIX, the Ice Thickness Models Intercomparison eXperiment

Abstract. Knowledge of the ice thickness distribution of glaciers and ice caps is an important prerequisite for many glaciological and hydrological investigations. A wealth of approaches has recently been presented for inferring ice thickness from characteristics of the surface. With the Ice Thickness Models Intercomparison eXperiment (ITMIX) we performed the first coordinated assessment quantifying individual model performance. A set of 17 different models showed that individual ice thickness estimates can differ considerably – locally by a spread comparable to the observed thickness. Averaging the results of multiple models, however, significantly improved the results: on average over the 21 considered test cases, comparison against direct ice thickness measurements revealed deviations on the order of 10 ± 24 % of the mean ice thickness (1σ estimate). Models relying on multiple data sets – such as surface ice velocity fields, surface mass balance, or rates of ice thickness change – showed high sensitivity to input data quality. Together with the requirement of being able to handle large regions in an automated fashion, the capacity of better accounting for uncertainties in the input data will be a key for an improved next generation of ice thickness estimation approaches.

Ensemble forecasting of Corbicula fluminea worldwide distribution: Projections of the impact of climate change

Abstract Global biodiversity is at risk owing to climate change, and freshwater ecosystems are expected to suffer the most. In recent years niche‐based models (NBMs) have been used to predict species distribution and are an important tool for conservation and management of aquatic ecosystems. In this work, the current and future climatic suitability areas of the invasive species Corbicula fluminea, which has known adverse ecological and economic impacts, were investigated. The species distribution modelling was based on nine algorithms in BIOMOD2, summarized in an ensemble forecasting approach. To model the species distribution, eight climatic parameters related to temperature and precipitation variables were considered. Three time frames (current, 2050 and 2070) were modelled using four increasing CO2 emission scenarios. The performance of individual models was excellent according to the area under the receiver operating characteristic curve (AUC) and good to excellent according to true skill statistics (TSS). Annual mean temperature, minimum temperature of the coldest month and mean temperature of the coldest quarter were the most important variables predicting C. fluminea occurrence. Of the total continental area, 6.6% was predicted to be suitable for C. fluminea in current conditions. In the future, suitable area will increase from the current value of 6.6% to values from 9.4% to 12.6%, according to the 2050 projections and up to 12.7% in 2070 in high emission scenarios. Overall, the results indicate that climate change will favour the expansion of C. fluminea into new river basins, especially at higher latitudes, and that future climatic scenarios may double the suitable area for Corbicula fluminea.

Capacity of semi-parametric regression models to predict extreme-event water quality in the Northeastern US

This study assessed the capacity of semi-parametric regression models to predict riverine solute concentrations during extreme high-flow hydrologic events, when such events are absent from the models' calibration data. Using a large dataset from 459 monitoring stations across the US Northeast, the models showed a tendency to overpredict extreme-event concentrations, with increasing bias and variance for increasingly extreme hydrologic conditions. The validation framework in this study effectively compared model performance across disparate hydrologic regimes and constituents, yet can be used to estimate individual model performance under an unobserved extreme-flow condition, regardless of whether any extreme-flow data are available for that model. The validation procedure can further be generalized to explore model performance in an arbitrarily defined extreme condition for a broad range of model types. Despite an overall increase in uncertainty for extreme-event concentration estimates, estimates under extreme hydrologic conditions could be improved by taking into account the observed bias in the aggregated regional database.

Short-run price forecast performance of individual and composite models for 496 corn cash markets

ABSTRACTUsing daily prices from 496 corn cash markets for July 2006–February 2011, this study investigates short-run forecast performance of 31 individual and 10 composite models for each market at horizons of 5, 10, and 30 days. Over the performance evaluation period September 2010–February 2011, two composite models are optimal across horizons for different markets based on the mean-squared error. For around half of the markets at the horizon of 5 days and most of them at 10 and 30 days, the mean-squared error of a market's optimal model is significantly different from those of at least other 23 models evaluated for it. Root-mean-squared error reductions through switching from non-optimal models to the optimal are generally around 0.40%, 0.55%, and 0.87% at horizons of 5, 10, and 30 days.

Forecasting Inflation Using the Phillips Curve: Evidence from Swedish Data

This paper studies the forecasting ability of various Phillips curve specifications in a pseudo out-of-sample exercise for Swedish inflation over the period 1980–2014. Three measures of inflation are considered––headline inflation, underlying inflation, GDP deflator inflation, in addition to different activity variables, various econometric specifications and different sample periods. Although the results indicate heterogeneity in individual model performance and evidence of model instability, in general, the Phillips curve models improve inflation forecasts against the random walk benchmark for both headline inflation and underlying inflation, and fail to beat the random walk benchmark for GDP deflator inflation. Phillips curve forecasts beat the random walk benchmark especially for 2004–2013. The monetary regime change in 1993 from exchange rate targeting to inflation targeting is also taken into account. The results suggest that for all Phillips curve models and all three inflation measures, the performance of the Phillips curve depends on whether the data used for making the predictions was under the inflation targeting regime or not. Univariate forecasting models perform well for the fixed exchange rate regime period, but the Phillips curve models are useful under the inflation targeting regime.

Estimating Hourly Water Temperatures in Rivers Using Modified Sine and Sinusoidal Wave Functions

AbstractAir temperature is often used to estimate stream water temperature by developing regression models. A method using modified sine functions and sinusoidal wave functions (MSSWF) was proposed for estimating hourly water temperatures in rivers. Estimates of daily maximum and minimum water temperatures from sine functions (SFs) were corrected using linear regressions as functions of deviations of estimates of daily maximum and minimum air temperatures from SFs, and then modified sinusoidal wave functions (SWFs) were used to estimate hourly water temperatures. Excellent agreement was found between observed and estimated hourly water temperatures using MSSWF models developed for 13 rivers in Alabama. The average Nash-Sutcliffe efficiency (NSE) for the MSSWF models developed for individual rivers is 0.94. The distance of a river monitoring station from the weather station has very little effect on the performance of individual MSSWF models when the distance is less than approximately 300 km. A lumped MSS...

Big Data Analytics for Prostate Radiotherapy.

Radiation therapy is a first-line treatment option for localized prostate cancer and radiation-induced normal tissue damage are often the main limiting factor for modern radiotherapy regimens. Conversely, under-dosing of target volumes in an attempt to spare adjacent healthy tissues limits the likelihood of achieving local, long-term control. Thus, the ability to generate personalized data-driven risk profiles for radiotherapy outcomes would provide valuable prognostic information to help guide both clinicians and patients alike. Big data applied to radiation oncology promises to deliver better understanding of outcomes by harvesting and integrating heterogeneous data types, including patient-specific clinical parameters, treatment-related dose–volume metrics, and biological risk factors. When taken together, such variables make up the basis for a multi-dimensional space (the “RadoncSpace”) in which the presented modeling techniques search in order to identify significant predictors. Herein, we review outcome modeling and big data-mining techniques for both tumor control and radiotherapy-induced normal tissue effects. We apply many of the presented modeling approaches onto a cohort of hypofractionated prostate cancer patients taking into account different data types and a large heterogeneous mix of physical and biological parameters. Cross-validation techniques are also reviewed for the refinement of the proposed framework architecture and checking individual model performance. We conclude by considering advanced modeling techniques that borrow concepts from big data analytics, such as machine learning and artificial intelligence, before discussing the potential future impact of systems radiobiology approaches.

Multimodel Prediction of Monsoon Rain Using Dynamical Model Selection

In this paper, dynamical-model-selection-based multimodel ensemble (DMS-MME) technique is developed for skill improvement of monsoon rain prediction in the medium range (i.e., 24-120 h ahead). The data set consists of 24-120 h daily precipitation forecasts from five state-of-the-art global circulation models (GCMs), i.e., European Centre for Medium Range Weather Forecasts (Europe), National Center for Environmental Prediction (USA), China Meteorological Administration (China), Canadian Meteorological Centre (Canada) and U.K. Meteorological Office (U.K.). The DMS-MME forecasts are constructed during the monsoon months (JJAS) for the years 2008-13 over the Indian mainland. For the training and verification purposes, India Meteorological Department rainfall is used. The forecast skill of the DMS-MME model has been compared with the performance of individual models and regression-based MME model. Further, to remove the nonnormality of rainfall distribution, square-root and logarithmic transfer functions are used for normalizing the precipitation data. The impact of these transfer functions on the forecast skill of the DMS-MME model has been assessed. The forecast skill of the proposed model is evaluated using the standard statistical measures. DMS-MME forecasts carries higher skill in terms of verification scores compared with the MME forecasts up to 120 h. It has been found that using the DMS-MME approach with a square-root transfer function (SDMS-MME) gives the best results. SDMS-MME outperforms the operational models and the regression-based MME at all forecast steps.

A hybrid evolutionary decomposition system for time series forecasting

In the field of time series forecasting, the combination of linear and nonlinear models has been explored to improve the accuracy of predictions when compared with the performance of individual models. Traditional forecasting methods such as the autoregressive integrated moving average (ARIMA) and exponential smoothing (ETS) are employed to map linear patterns in a time series while neural networks (ANNs) and support vector machines (SVMs) map nonlinear patterns. A traditional hybrid ARIMA–ANN system works by performing linear relationships in the data, considering the residual error a nonlinear component which is mapped by the ANN. The nature of a time series can be taken into consideration before applying a model. This work employs both linear and nonlinear patterns of a time series based on its volatility. This is explored using a hybrid evolutionary system composed by a simple exponential smoothing filter, ARIMA and autoregressive (AR) linear models and a SVR model. Particle swarm optimization is employed to optimize the order of the AR model, SVR parameters and number of lags of the time series. Experimental results show that the evolutionary hybrid system presented promising results in the forecasting domain.

Effective soil moisture estimate and its uncertainty using multimodel simulation based on Bayesian Model Averaging

AbstractVarious hydrological models have been developed for estimating root zone soil moisture dynamics. These models, however, incorporated their own parameterization approaches indicating that the output from the different model inherent structures might include uncertainties because we do not know which model structure is correct for describing the real system. More recently, multimodel approaches using a Bayesian Model Averaging (BMA) scheme can improve the overall predictive skill while individual models retain their own uncertainties for simulated soil moisture based on a single set of weights in modeling under different land surface wetness conditions (e.g., wet, moderately wet, and dry conditions). In order to overcome their limitations, we developed a BMA‐based multimodel simulation approach based on various soil wetness conditions for estimating effective surface soil moisture dynamics (0–5 cm) and quantifying uncertainties efficiently based on the land surface wetness conditions. The newly developed approach adapts three different hydrological models (i.e., Noah Land Surface Model, Noah LSM; Soil‐Water‐Atmosphere‐Plant, SWAP; and Community Land Model, CLM) for simulating soil moisture. These models were integrated with a modified‐microGA (advanced version of original Genetic Algorithm (GA)) to search for optimized soil parameters for each model. Soil moisture was simulated from the estimated soil parameters using the hydrological models in a forward mode. It was found that SWAP performed better than others during wet condition, while Noah LSM and CLM showed a good agreement with measurements during dry condition. Thus, we inferred that performance of individual models with different model structures can be different with land surface wetness. Taking into account the effects of soil wetness on different model performances, we categorized soil moisture measurements and estimated different weights for each category using the BMA scheme. Effective surface soil moisture dynamics were obtained by aggregating multiple weighted soil moisture. Our findings demonstrated that the effective soil moisture estimates derived by this approach showed a better match with the measurements compared to the original models and single‐weighted outputs. Multimodel simulation approach based on land surface wetness enhances the ability to predict reliable soil moisture dynamics and reflects the strengths of different hydrological models under various soil wetness conditions.

Evaluating the skill of NMME seasonal precipitation ensemble predictions for 17 hydroclimatic regions in continental China

There is an increasing focus on the usefulness of climate model–based seasonal precipitation forecasts as inputs for hydrological applications. This study reveals that most models from the North American Multi-Model Ensemble (NMME) have potential to forecast seasonal precipitation over 17 hydroclimatic regions in continental China. In this paper, we evaluated the NMME precipitation forecast against observations. The evaluation indices included the correlation coefficient (R), relative root-mean-square error (RRMSE), rank histogram (RH), and continuous ranked probability skill score (CRPSS). We presented the RRMSE-R diagram to distinguish differences between the performances of individual models. We find that the predictive skill is seasonally and regionally dependent, exhibiting higher values in autumn and spring and lower values in summer. Higher predictive skill is observed over most regions except the southeastern monsoon regions, which may be attributable to local climatology and variability. Among the 11 NMME models, CFS, especially CFSv2, exhibits the best predictive skill. The GFDL and NASA models, which are followed by CMC, perform worse than CFS. The performances of IRI and CCSM3 are relatively worse than that of the other models. The forecast skills are significantly improved in multi-model mean forecasts based on simple model averaging (SMA). The improvement is more obvious for Bayesian model averaging (BMA), which is employed to further improve the forecast skill and address model uncertainty using multiple model outputs, than individual model and SMA.

Prediction of subacute ruminal acidosis based on milk fatty acids: A comparison of linear discriminant and support vector machine approaches for model development

Subacute ruminal acidosis (SARA), characterized by low rumen pH, is one of the most important metabolic disorders in dairy cattle. As dairy cows experiencing SARA often do not exhibit overt clinical symptoms, diagnostic biomarkers in milk are of interest. Data of six acidosis induction experiments with rumen-fistulated dairy cows were combined to assess the potential of milk fatty acids (FA) to identify acidotic cases, based on three threshold values often reported in literature, i.e. time pH<5.6 of 180min/d and 283min/d and time pH below 5.8 of 475min/d (N=442 cases, of which 111–165 acidotic cases, depending on the applied threshold value). Both linear discriminant analysis (LDA) as well as support vector machines (SVM) were used to develop classification models, with SVM based on two common types of kernel functions (linear kernels and Gaussian radial basis function kernels) and including either the whole milk FA profile (41–69 milk FA, depending on the experiment) or a selected number of milk FA (i.e. both odd and branched chain FA and biohydrogenation derivates of poly-unsaturated FA, 13–16 FA). Both evaluation of the performance of individual classification models as well as comparison of models was based on the area under the receiver operating characteristic (ROC) curve. Non-linear models developed through a radial kernel based SVM approach seemed of particular interest when including all milk FA as model features. However, linear models based on the selected group of milk FA most often performed as good as the non-linear models including all milk FA, with the former being least time consuming and more cost-effective, both from a computational as well as an analytical perspective. However, combination of all data sets only resulted in good classification models when including data of each dataset upon training the model, whereas model performance decreased dramatically in case of cross-dataset cross-validation. This indicates an important impact of the origin of the datasets on the performance of the model which should be taken into account in further exploration of prediction models of SARA.

A novel ensemble scheme for the multi-focus image fusion using support vector machine

In the multi-focus image fusion problem, the source images are obtained from the same scene. They are fused to get an image that contains all well-focussed objects. Previously, individual machine-learning models are proposed for image fusion. The performance of individual models is limited to fuse the useful information extracted from the blurred images. To address this problem, we developed a novel ensemble scheme for multi-focus image fusion using support vector machines (SVMs). In the proposed scheme, first, SVM models are constructed using different kernel functions of linear, polynomial, radial basis, and sigmoid. The predictions of individual SVM models are then combined using majority voting. In this way, the combined decision space becomes more informative and discriminant. A comparative analysis of the proposed scheme is carried out with previous techniques. It is found that our scheme is more accurate for synthesized-blurred and real defocussed images.

Climate extremes indices in the CMIP5 multimodel ensemble: Part 1. Model evaluation in the present climate

This paper provides a first overview of the performance of state‐of‐the‐art global climate models participating in the Coupled Model Intercomparison Project Phase 5 (CMIP5) in simulating climate extremes indices defined by the Expert Team on Climate Change Detection and Indices (ETCCDI), and compares it to that in the previous model generation (CMIP3). For the first time, the indices based on daily temperature and precipitation are calculated with a consistent methodology across multimodel simulations and four reanalysis data sets (ERA40, ERA‐Interim, NCEP/NCAR, and NCEP‐DOE) and are made available at the ETCCDI indices archive website. Our analyses show that the CMIP5 models are generally able to simulate climate extremes and their trend patterns as represented by the indices in comparison to a gridded observational indices data set (HadEX2). The spread amongst CMIP5 models for several temperature indices is reduced compared to CMIP3 models, despite the larger number of models participating in CMIP5. Some improvements in the CMIP5 ensemble relative to CMIP3 are also found in the representation of the magnitude of precipitation indices. We find substantial discrepancies between the reanalyses, indicating considerable uncertainties regarding their simulation of extremes. The overall performance of individual models is summarized by a “portrait” diagram based on root‐mean‐square errors of model climatologies for each index and model relative to four reanalyses. This metric analysis shows that the median model climatology outperforms individual models for all indices, but the uncertainties related to the underlying reference data sets are reflected in the individual model performance metrics.

Evaluation of different models to estimate the global solar radiation on inclined surfaces

Abstract Global and diffuse solar radiation intensities are, in general, measured on horizontal surfaces, whereas stationary solar conversion systems (both flat plate solar collector and solar photovoltaic) are mounted on inclined surface to maximize the amount of solar radiation incident on the collector surface. Consequently, the solar radiation incident on a tilted surface has to be determined by converting solar radiation from horizontal surface to the tilted surface of interest. This study evaluates the performance of 14 models transposing 10 min diffuse solar irradiation from horizontal to inclined surface. Solar radiation data from 8 months (April to November 2011) which include diverse atmospheric conditions and solar altitudes, measured on the roof of the radiation tower of the Royal Meteorological Institute of Belgium in Uccle (Longitude 4.35° E, Latitude 50.79° N) were used for validation purposes. Individual model performance is assessed by an inter-comparison between the calculated and measured solar global radiation on a south-oriented surface tilted at 50.79° using statistical methods. The relative performance of the different models under different sky conditions has been studied. Because statistical validation procedures revealed that none of the considered model performs well under all types of sky conditions a new model resulting from the coupling of three models acting under different sky conditions, is developed for Belgium. The sensitivity of the proposed model to the ground reflection formulation is assessed. Finally, the ability of the coupled model to handle hourly and daily data is discussed.

Dynamical linkage of tropical and subtropical weather systems to the intraseasonal oscillations of the Indian summer monsoon rainfall. Part II: Simulations in the ENSEMBLES project

We assess the ability of individual models (single-model ensembles) and the multi-model ensemble (MME) in the European Union-funded ENSEMBLES project to simulate the intraseasonal oscillations (ISOs; specifically in 10–20-day and 30–50-day frequency bands) of the Indian summer monsoon rainfall (ISMR) over the Western Ghats (WG) and the Bay of Bengal (BoB), respectively. This assessment is made on the basis of the dynamical linkages identified from the analysis of observations in a companion study to this work. In general, all models show reasonable skill in simulating the active and break cycles of the 30–50-day ISOs over the Indian summer monsoon region. This skill is closely associated with the proper reproduction of both the northward propagation of the intertropical convergence zone (ITCZ) and the variations of monsoon circulation in this band. However, the models do not manage to correctly simulate the eastward propagation of the 30–50-day ISOs in the western/central tropical Pacific and the eastward extension of the ITCZ in a northwest to southeast tilt. This limitation is closely associated with a limited capacity of models to accurately reproduce the magnitudes of intraseasonal anomalies of both the ITCZ in the Asian tropical summer monsoon regions and trade winds in the tropical Pacific. Poor reproduction of the activity of the western Pacific subtropical high on intraseasonal time scales also amplify this limitation. Conversely, the models make good reproduction of the WG 10–20-day ISOs. This success is closely related to good performance of the models in the representation of the northward propagation of the ITCZ, which is partially promoted by local air–sea interactions in the Indian Ocean in this higher-frequency band. Although the feature of westward propagation is generally represented in the simulated BoB 10–20-day ISOs, the air–sea interactions in the Indian Ocean are spuriously active in the models. This leads to active WG rainfall, which is not present in the observed BoB 10–20-day ISOs. Further analysis indicates that the intraseasonal variability of the ISMR is generally underrepresented in the simulations. Skill of the MME in seasonal ISMR forecasting is strongly dependent on individual model performance. Therefore, in order to improve the model skill with respect to seasonal ISMR forecasting, we suggest it is necessary to better represent the robust dynamical links between the ISOs and the relevant circulation variations, as well as the proportion of intraseasonal variability in the individual models.

The Continual Intercomparison of Radiation Codes: Results from Phase I

We present results from Phase I of the Continual Intercomparison of Radiation Codes (CIRC), intended as an evolving and regularly updated reference source for evaluation of radiative transfer (RT) codes used in global climate models and other atmospheric applications. CIRC differs from previous intercomparisons in that it relies on an observationally validated catalog of cases. The seven CIRC Phase I baseline cases, five cloud free and two with overcast liquid clouds, are built around observations by the Atmospheric Radiation Measurements program that satisfy the goals of Phase I, namely, to examine RT model performance in realistic, yet not overly complex, atmospheric conditions. Besides the seven baseline cases, additional idealized “subcases” are also employed to facilitate interpretation of model errors. In addition to quantifying individual model performance with respect to reference line‐by‐line calculations, we also highlight RT code behavior for conditions of doubled CO2, issues arising from spectral specification of surface albedo, and the impact of cloud scattering in the thermal infrared. Our analysis suggests that improvements in the calculation of diffuse shortwave flux, shortwave absorption, and shortwave CO2 forcing as well as in the treatment of spectral surface albedo should be considered for many RT codes. On the other hand, longwave calculations are generally in agreement with the reference results. By expanding the range of conditions under which participating codes are tested, future CIRC phases will hopefully allow even more rigorous examination of RT codes.

Toward reduction of model uncertainty: Integration of Bayesian model averaging and data assimilation

Multimodeling in hydrologic forecasting has proved to improve upon the systematic bias and general limitations of a single model. This is typically done by establishing a new model as a linear combination or a weighted average of several models with weights on the basis of individual model performance in previous time steps. The most commonly used multimodeling method, Bayesian model averaging (BMA), assumes a fixed probability distribution around individual models' forecast in establishing the prior and uses a calibration period to determine static weights for each individual model. More recent work has focused on a sequential Bayesian model selection technique with weights that are adjusted at each time step in an attempt to accentuate the dynamics of an individual model's performance with respect to the system's response. However, these approaches still assume a fixed distribution around the individual models' forecast. A new sequential Bayesian model‐averaging technique is developed incorporating a sliding window of individual model performance around the forecast. Additionally, this new technique relaxes the fixed distribution assumption in establishing the prior utilizing a particle filter data assimilation method that reflects both the performance dynamics of the models' forecasts along with their uncertainty. A comparative analysis of the different BMA strategies is performed across different rates of change in the hydrograph. Results show that methods employing the particle filter show higher probabilistic skill in high ranges of volatility but are overconfident in medium and low ranges of volatility.

Air quality simulations for North America - MM5–CAMx modelling performance for main gaseous pollutants

Abstract In the scope of the Air Quality Model Evaluation International Initiative (AQMEII) the air quality modelling system MM5–CAMx was applied to the North American (NA) domain for calendar year 2006. The simulation domain was defined according to the spatial resolution and the coordinate system of the emission databases provided and the common grid required by AQMEII for ensemble analysis. A Lambert Conformal Projection grid of around 5500 km by 3580 km with 24 × 24 km2 horizontal resolution was defined. Emissions available through AQMEII have been prepared to feed the CAMx model. Meteorological inputs were developed by the application of the meteorological model MM5, which was initialized by 1° resolution NCEP-FNL global data and run for the whole year of 2006. A spatial and temporal analysis of results based on the 2D surface fields and time series for regional monitoring stations was performed for the main gaseous pollutants. A detailed statistical analysis and evaluation against observations was carried out, considering three different sub-domains over North America, in order to comprehend the differences between the East, West and Central part. The exploitation of modelling results was based on the capabilities and analysis tools available through the ENSEMBLE software, developed and upgraded for AQMEII. Results have shown a good agreement between observed and modelled concentrations of O3 (especially regarding peaks) and NO2 and a weaker performance of the air quality model for CO and SO2. However, the model tends to underestimate O3 and overestimate NO2 and CO at night as a consequence of meteorology (weak vertical mixing due to underestimation of the Planetary Boundary Layer (PBL) height). This paper intends to be a valuable contribution to the overall AQMEII exercise since it aims to evaluate the performance of individual models to be used in the ensemble approach for the areas of interest.

Considerations in the Selection of Global Climate Models for Regional Climate Projections: The Arctic as a Case Study*

Abstract Climate projections at regional scales are in increased demand from management agencies and other stakeholders. While global atmosphere–ocean climate models provide credible quantitative estimates of future climate at continental scales and above, individual model performance varies for different regions, variables, and evaluation metrics—a less than satisfying situation. Using the high-latitude Northern Hemisphere as a focus, the authors assess strategies for providing regional projections based on global climate models. Starting with a set of model results obtained from an “ensemble of opportunity,” the core of this procedure is to retain a subset of models through comparisons of model simulations with observations at both continental and regional scales. The exercise is more one of model culling than model selection. The continental-scale evaluation is a check on the large-scale climate physics of the models, and the regional-scale evaluation emphasizes variables of ecological or societal relevance. An additional consideration is given to the comprehensiveness of processes included in the models. In many but not all applications, different results are obtained from a reduced set of models compared to relying on the simple mean of all available models. For example, in the Arctic the top-performing models tend to be more sensitive to greenhouse forcing than the poorer-performing models. Because of the mostly unexplained inconsistencies in model performance under different selection criteria, simple and transparent evaluation methods are favored. The use of a single model is not recommended. For some applications, no model may be able to provide a suitable regional projection. The use of model evaluation strategies, as opposed to relying on simple averages of ensembles of opportunity, should be part of future synthesis activities such as the upcoming Fifth Assessment Report of the Intergovernmental Panel on Climate Change.