Evaluation Of Environmental Models Research Articles

Technical assessment and evaluation of environmental models and software: Letter to the Editor

This letter details the collective views of a number of independent researchers on the technical assessment and evaluation of environmental models and software. The purpose is to stimulate debate and initiate action that leads to an improved quality of model development and evaluation, so increasing the capacity for models to have positive outcomes from their use. As such, we emphasize the relationship between the model evaluation process and credibility with stakeholders (including funding agencies) with a view to ensure continued support for modelling efforts.Many journals, including EM&S, publish the results of environmental modelling studies and must judge the work and the submitted papers based solely on the material that the authors have chosen to present and on how they present it. There is considerable variation in how this is done with the consequent risk of considerable variation in the quality and usefulness of the resulting publication. Part of the problem is that the review process is reactive, responding to the submitted manuscript. In this letter, we attempt to be proactive and give guidelines for researchers, authors and reviewers as to what constitutes best practice in presenting environmental modelling results. This is a unique contribution to the organisation and practice of model-based research and the communication of its results that will benefit the entire environmental modelling community. For a start, our view is that the community of environmental modellers should have a common vision of minimum standards that an environmental model must meet. A common vision of what a good model should be is expressed in various guidelines on Good Modelling Practice. The guidelines prompt modellers to codify their practice and to be more rigorous in their model testing. Our statement within this letter deals with another aspect of the issue – it prompts professional journals to codify the peer-review process. Introducing a more formalized approach to peer-review may discourage reviewers from accepting invitations to review given the additional time and labour requirements. The burden of proving model credibility is thus shifted to the authors. Here we discuss how to reduce this burden by selecting realistic evaluation criteria and conclude by advocating the use of standardized evaluation tools as this is a key issue that needs to be tackled.

Ten steps applied to development and evaluation of process-based biogeochemical models of estuaries

The procedures involved in model development may be set out as a ten step process, beginning with defining the purpose of the model and ending with evaluation of the appropriateness and utility of the completed model. This process, recently outlined by Jakeman et al. [Jakeman, A.J., Letcher, R.A., Norton, J.P., 2006. Ten iterative steps in development and evaluation of environmental models. Environmental Modelling and Software 21, 602–614], is often iterative as model development is a continuous process that refines and improves the intended capacity of the model. Here, the ten steps of model development are critiqued and applied using a process-based biogeochemical model of aquatic systems, with examples from two case studies: a model of phytoplankton succession and nutrient concentrations in the Swan-Canning Estuary (Western Australia) and a model of sediment and nutrient transport and transformation in the Fitzroy Estuary and Keppel Bay (Queensland).

Water balance modelling in Bowen, Queensland, and the ten iterative steps in model development and evaluation

Jakeman et al. (Jakeman, A.J., Letcher, R.A., Norton, J.P., 2006. Ten iterative steps in development and evaluation of environmental models. Environmental Modelling & Software 21, 602–614) outline ten steps in the pursuit of good practice in model development and application to increase the credibility and impact of results from environmental models. This paper shows how the ten steps of model development are relevant to numerical groundwater modelling, using a model of a data-rich coastal groundwater system near Bowen in Queensland, Australia as an example. The model is Geographic Information System-based and estimates the dynamical water balance using Darcy's Law. The method, which is generally applicable to data-rich aquifers, proved cost and time effective and provided important insights to the groundwater dynamics of the area.

An integrated multi-model approach for air quality assessment: Development and evaluation of the OSCAR Air Quality Assessment System

This paper reports on the development and evaluation of a new modelling system for studying air quality on local scales. A multi-model approach has been adopted to develop the OSCAR Air Quality Modelling System to conduct assessments at different levels of complexity. This work focuses on the evaluation of the performance of two of the models, CAR II (semi-empirical, annual averaged model) and CAR-FMI (line-source Gaussian model for hourly predictions), as integrated into the OSCAR System and employing consistent emissions and meteorological inputs. The models have been evaluated using measured NO x, NO 2, PM 10 and PM 2.5 data for 2003 from the Vallila (Helsinki) and Cromwell Road (London) monitoring stations; both of these stations are located in urban roadside environments. The evaluation consists of comparisons of annual averaged and hourly predicted and measured data, as well as detailed statistical analysis of the model performance. Overall, closer agreement between the modelled and measured data is observed for the Vallila site, where traffic levels are medium, but the models show significant underestimation for Cromwell Road which is subject to high traffic levels. The paper also discusses possible reasons for this underestimation. The wider implications of the work are also stated for development and evaluation of environmental models.

Ten iterative steps in development and evaluation of environmental models

Models are increasingly being relied upon to inform and support natural resource management. They are incorporating an ever broader range of disciplines and now often confront people without strong quantitative or model-building backgrounds. These trends imply a need for wider awareness of what constitutes good model-development practice, including reporting of models to users and sceptical review of models by users. To this end the paper outlines ten basic steps of good, disciplined model practice. The aim is to develop purposeful, credible models from data and prior knowledge, in consort with end-users, with every stage open to critical review and revision. Best practice entails identifying clearly the clients and objectives of the modelling exercise; documenting the nature (quantity, quality, limitations) of the data used to construct and test the model; providing a strong rationale for the choice of model family and features (encompassing review of alternative approaches); justifying the techniques used to calibrate the model; serious analysis, testing and discussion of model performance; and making a resultant statement of model assumptions, utility, accuracy, limitations, and scope for improvement. In natural resource management applications, these steps will be a learning process, even a partnership, between model developers, clients and other interested parties.