Bayesian assessment of ecological footprint drivers in Finland: a model averaging approach under structural and model uncertainty

Large simulation models of eutrophication processes are commonly used to aid scientific understanding and to guide management decisions. Confidence in models for these purposes depends on uncertainty in model equations (structural uncertainty) and on effects of input uncertainties (model parameters, initial conditions, and forcing functions) on model outputs. Our objective herein is to illustrate two strategies, a generalized likelihood uncertainty estimation (GLUE) approach combined with a simple Monte Carlo sampling scheme and a Bayesian methodological framework along with Markov Chain Monte Carlo (MCMC) simulations, for elucidating the propagation of uncertainty in the high‐dimensional parameter spaces of mechanistic eutrophication models. We examine the ability of the two approaches to offer insights into the degree of information about model inputs that the data contain, to quantify the correlation structure among parameter estimates, and to obtain predictions along with uncertainty bounds for modeled output variables. Our analysis is based on a four‐state‐variable (phosphate‐detritus‐phytoplankton‐zooplankton) model and the mesotrophic Lake Washington (Washington State, United States) as a case study. Scientific knowledge, expert judgment, and observational data were used to formulate prior probability distributions and characterize the uncertainty pertaining to 14 model parameters. Despite the conceptual differences for addressing model equifinality, that is, wide ranges of parameter values subject to complex multivariate relationships that result in plausible observed behaviors and produce equivalently accurate predictions, we found that the two strategies provided fairly consistent estimates of the posterior parameter correlation structure and output uncertainty. Nonetheless, our analysis also shows that MCMC can more efficiently quantify the joint probability distribution of model parameters and make inference about this distribution. The latter finding can be explained by the basic idea underlying the MCMC methodology, that is, the configuration of a Markov process whose stationary distribution approximates the joint posterior distribution of all the stochastic model nodes; as a result, Monte Carlo samples are not drawn from the prior parameter space, and problems of wide or highly correlated prior distributions can be overcome. Finally, our study stresses the lack of perfect simulators of natural system dynamics and introduces two statistical formulations that can explicitly account for the discrepancy between mathematical models and environmental systems.

Managing wildlife populations for conservation, control or harvesting involves uncertainty. Nevertheless, decisions need to be made based on the available evidence. The two main sources of uncertainty in population modelling are parameter estimates and structural uncertainty. Structural uncertainty in models is not included as often as parameter uncertainty.We present an approach where parameter and structural uncertainty (strength of density dependence) is included within a model, using the over-wintering English population of cormorants Phalacrocorax carbo L. Because of the damage caused to inland fishery interests by cormorants, there was a change in UK government policy in autumn 2004, increasing the numbers of birds that can be shot under licence.A stochastic Monte Carlo annual population model was produced to examine the effect of changes to the numbers of birds shot each year. Indices of annual population size were converted to population estimates and used to determine annual growth rates and strength of density dependence.There is strong evidence for density dependence in the data, which suggests the population is currently slightly above carrying capacity, with a mean growth rate of 4–6% per annum. The 1300 birds shot under licence in 2004/05 represent about 4·5% of the English population, and if this level of culling continues, the population would be expected to decline by 9% by 2007, compared to the long-term average. The a priori preferred model, which included all uncertainty, gave predictions for 2004/05 and 2005/06 in close agreement with field data.The model was used to produce short-term population projections, with the understanding that Adaptive Resource Management (ARM) will be adopted to iteratively update the parameters and model each year, feeding back into the numbers of available licences.Synthesis and applications. We recommend the approach used in this study of including parameter and structural uncertainty within a single model, where possible, with the proportion of iterations that utilize a particular structure dependent on the weight of evidence for that structure. This will produce results with wider confidence intervals, but ensures that the evidence for any particular model is not over-interpreted.

Biometricians have made great strides in the generation of reliable estimates of demographic rates and their uncertainties from imperfect field data, but these estimates are rarely used to produce detailed predictions of the dynamics or future viability of at‐risk populations. Conversely, population viability analysis (PVA) modelers have increased the sophistication and complexity of their approaches, but most do not adequately address parameter and model uncertainties in viability assessments or include important ecological drivers. Merging the advances in these two fields could enable more defensible predictions of extinction risk and better evaluations of management options, but only if clear and interpretable PVA results can be distilled from these complex analyses and outputs. Here, we provide guidance on how to successfully conduct such a combined analysis, using the example of the endangered island fox ( Urocyon littoralis ), endemic to the Channel Islands of California, USA. This more rigorous demographic PVA was built by forming a close marriage between the statistical models used to estimate parameters from raw data and the details of the subsequent PVA simulation models. In particular, the use of mark–recapture analyses and other likelihood and information‐theoretic methods allowed us to carefully incorporate parameter and model uncertainty, the effects of ecological drivers, density dependence, and other complexities into our PVA. Island fox populations show effects of density dependence, predation, and El Niño events, as well as substantial unexplained temporal variation in survival rates. Accounting not only for these sources of variability, but also for uncertainty in the models and parameters used to estimate their strengths, proved important in assessing fox viability with different starting population sizes and predation levels. While incorporating ecological drivers into PVA assessments can help to predict realistic dynamics, we also show that unexplained process variance has important effects even in our extremely well‐studied system, and therefore must not be ignored in PVAs. Overall, the treatment of causal factors and uncertainties in parameter values and model structures need not result in unwieldy models or highly complex predictions, and we emphasize that future PVAs can and should include these effects when suitable data are available to support their analysis.

Chapter 8 - Quantifying uncertainty in environmental health models

Methodological issues of using observational human data in lung dosimetry models for particulates

To address the holistic aims of marine ecosystem-based management (EBM), ecosystem modelers seek to reflect the diverse processes, priorities, and perspectives of social-ecological systems to describe the equally complex systems we want to manage. Because of the multiple components, relationships, and parameters at play in models of marine ecosystems, structural (or model) uncertainty, which is uncertainty about 1) the number of modeled system components, 2) interactions between these components, and 3) the form and function of these relationships and values for parameters, can affect advice provided for EBM. Employing ecosystem or whole-of-system models in marine resource management presents a conflict between testing the performance of these models and structural uncertainty. Any model used as the basis for decision-making should first be shown to be robust to observation, process, and structural errors through tests of model performance with respect to relevant data. At the same time, test data become more and more scarce as additional processes and variables are included in these assessment models. Building from a conceptual model of the social-ecological system of Georges Bank, this dissertation explored these issues by 1) applying qualitative and Bayesian network modeling tools as simpler alternatives to complex whole-of-system models, 2) addressing model uncertainty in the Bayesian network model through a structural uncertainty framework, and 3) testing the performance of the qualitative and Bayesian network models for decision making through a management strategy evaluation applied at the ecosystem level. Chapter 1 applied qualitative network modeling to evaluate effects of structural uncertainty on management outcomes for twelve social, economic, and conservation objectives. This study compared the sensitivity of outcomes from two management strategies in four model structures of the Georges Bank system that investigate uncertainty in trophic structure and fisheries configuration. This analysis revealed that the resolution used to represent fisheries affects perceived tradeoffs among objectives as well as the reliability of predictions. Chapter 2ivdeveloped and fit a Bayesian network of the Georges Bank system to a 58-year time series of observations and expert information. With a focus on predictive ability of the model, this analysis tested the accuracy of the Bayesian network using within-sample and simulated datasets and measured the sensitivity of model predictions to additional information. The model predicted system states with 73% and 68% accuracy one and two years in advance, respectively, but some indicators were predicted more accurately than others. Chapter 3 demonstrated a framework for assessing effects of structural uncertainty on perceived management objective outcomes by comparing predictions from nine alternate Bayesian network model structures. After applying four management scenarios, differences among model alternatives were summarized from two performance perspectives: 1) optimization of estimated utility and 2) satisficing, or meeting a minimum threshold, for a maximum number of objectives. This structural sensitivity analysis showed that tradeoffs among objectives may be affected by structural uncertainty, contingent on the performance metric(s) used to make decisions. In Chapter 4, multi-model comparison and simulation testing emulating management strategy evaluation were used to evaluate the performance of the qualitative and Bayesian network models developed in Chapters 1 and 2 in an EBM decision context. Performance testing of these models found that 1) resolution of the qualitative network model may be too coarse to adequately represent the Georges Bank system for evaluating multiple cumulative pressures and 2) the Bayesian network model formulated in Chapter 2 may mischaracterize predictions of system states that are important for guiding strategic EBM decisions. Nonetheless, this case study reveals modeling approaches that address model structure that may improve performance of the Georges Bank Bayesian network model. This dissertation demonstrates that addressing structural uncertainty in whole-of-system models is essential for providing EBM advice and outlines a targeted approach to evaluate its effects using a range of qualitative and quantitative modeling tools.

<p>A hydrological model incurs three types of uncertainties: measurement, structural and parametric uncertainty. Measurement uncertainty exists due to errors in the measurements of rainfall and streamflow data. Structural uncertainty exists due to errors in the mathematical representation of hydrological processes. Parametric uncertainty is a consequence of limited data available to calibrate the model, and measurement and structural uncertainties.</p><p>Recently, separation of structural and measurement uncertainties was identified as one of the twenty-three unsolved problems in hydrology. The information about measurement and structural uncertainties is typically available in the form of residual time-series, that is, the difference between observed and simulated streamflow time-series. The residual time-series, however, provides only an aggregate measure of measurement and structural uncertainties. Thus, the measurement and structural uncertainties are inseparable without additional information. In this study, we used random forest (RF) algorithm to gather additional information about measurement uncertainties using hydrological data across several watersheds. Subsequently, the uncertainty bounds obtained by RF were compared against the uncertainty bounds obtained by two other methods: rating-curve analysis and recently proposed runoff-coefficient method. Rating curve analysis yields uncertainty in streamflow measurements only and the runoff-coefficient yields uncertainty in both rainfall and streamflow measurements. The results of the study are promising in terms of using data across different watersheds for the construction of measurement uncertainty bounds. The preliminary results of this study will be presented in the meeting.</p>

Aeroelastic flutter analysis considering modeling uncertainties

Physics-based computer models, such as fluid flow simulations, seek to approximate the behavior of a real system based on the physical equations that govern the evolution of that system. The model approximation of reality, however, is imperfect because it is subject to uncertainties coming from different sources: finite model resolution, uncertainty in model parameter values, uncertainty in input data such as external forgings, and uncertainty in the structure of the model itself. Many studies to date have considered the effects of parameter and data uncertainty on model outputs, and have offered solutions to obtain the best fitted parameter values for a model. However, much less effort has been devoted to the study of structural uncertainty, which is caused by our incomplete knowledge about the true physical processes, and manifests itself as missing dynamics in the model. This paper seeks to understand structural uncertainty by studying the observable errors, i.e. the discrepancies between the model solutions and measurements of the physical system. The dynamics of these errors is modeled using a state-space approach, which enables one to identify the source of uncertainty and to recognize the missing dynamics inside model. Furthermore, the model solution can be improved by correcting it with the error predicted by the state-space approach. The proposed methodology is applied to two test problems, Lorenz-96 and a stratospheric chemistry model.

Uncertainty in simulating water resource systems (WRSs) makes it difficult to assess how effective different water management decisions will be. Uncertainty in simulation models can undermine the credibility of simulation and optimization studies and the uptake of their results. We identify different sources of uncertainty in WRS models and find that structural uncertainty (i.e., around definition of interrelationships within the system) and contextual uncertainty (i.e., around definition of the system boundaries) are rarely considered when simulating and optimizing WRSs. We propose a methodology to quantify the effects of structural and contextual uncertainties on the estimated performance of optimized water management decisions and demonstrate that they have a significant impact on a real‐world case study of a pumped‐storage system in the UK. To the best of the authors' knowledge, this is the first study to consider the impact of these types of uncertainty on optimized operating policies and their simulated performances. Our main finding is that of all the considered uncertainties, the assumptions made about context—specifically around the level of cooperation between neighboring water companies—had the greatest impact on performance estimates. This is important because few WRSs exist in isolation, yet discussion of the effects that a given definition of the system boundaries have on the simulation/optimization results is uncommon. We also highlight the significance of adequately considering aleatory uncertainty when evaluating performance estimates—something that few studies do—and present a simple technique to justify the sample size used for the evaluation of optimization results.

Hydrologic and water quality models (HWQMs) are increasingly used to support decisions on various environmental issues and policy directions for present and future scenarios, at scales varying from watershed to continental levels. Uncertainty associated with such models may affect the ability of the models to accurately evaluate the response of complex systems, leading to misguided assessments and risk management decisions. Current well-known HWQMs contain numerous input parameters, many of which are not known with certainty, and in other cases model users can hardly recognize the genesis of uncertainty. Uncertainty in data, model structure, and model parameters can propagate throughout model runs, causing the model output to substantially deviate from the expected response of the natural system. Various uncertainty assessment methods have been used with different HWQMs, creating concerns about an adequate approach for handling uncertainty in these models and how such an approach can be implemented across various discretization complexities and scales. In this article, our primary intention is to review uncertainty in the currently used HWQMs and to provide guidance and useful information for researchers and investigators. In this regard, we explore the genesis of uncertainty in hydrologic and water quality modeling (i.e., spatiotemporal scales, model representation, model discretization, model parameterization) and provide strategies for assessing uncertainty in hydrologic and water quality modeling on local and global scales when interpreting the model output.

<p>To quantify risk from natural hazards and ensure a robust decision-making process in the insurance industry, uncertainties in the mathematical models that underpin decisions need to be efficiently and robustly captured. The complexity and sheer scale of the mathematical modelling often makes a comprehensive, transparent and easily communicable understanding of the uncertainties very difficult.  Models predicting flood hazard and risk have shown high levels of uncertainty in their predictions due to data limitations and model structural uncertainty. Moreover, uncertainties are estimated to increase with climate change, especially for higher warming levels.</p><p>Global Sensitivity Analysis (GSA) provides a structured approach to quantify and compare the relative importance of parameter, data and structural uncertainty. GSA has been implemented successfully in tools such as the Sensitivity Analysis For Everybody (SAFE) toolbox, which is currently used by more than 2000 researchers worldwide. However, tailored tools, workflows and case studies are needed to demonstrate GSA benefits to practitioners and accelerate its uptake by the insurance industry.</p><p>One such case study has been the collaboration between the University of Bristol and JBA Risk Management on JBA’s new Global Flood Model, whose technology and flexibility has allowed to test a catastrophe model in ways not possible in the past. JBA has gained great insight into the sensitivity of modelled losses to uncertainties in the model datasets and analysis options. This has helped to explore the key sensitivities of the results to the assumptions made, for example to visualise how the distribution of modelled losses varies by return period and explore which parameters have the biggest impact on loss for the part of the Exceedance-Probability curve of interest. This information is essential for insurance companies to form their view of risk and to empower model users to adequately communicate uncertainties to decision-makers.</p>

Model uncertainty is a relatively new topic of discussion in TH code calculations, despite being often the major contributor to the overall uncertainty and a challenging practice in uncertainty analysis. The Integrated thermal-hydraulics uncertainty analysis (IMTHUA) methodology, developed by the authors, treats the TH code structural uncertainties (generally known as model uncertainty) explicitly by treating internal sub-model uncertainties, and by propagating such model uncertainties in the code calculations, including uncertainties about input parameters. This paper presents systematic model uncertainty of thermal-hydraulics system codes as part of IMTHUA methodology. The objective is to demonstrate effectiveness and practicality of the methodology on complex thermal-hydraulics system codes calculations and discuss the challenges dealing with these types of uncertainty sources. TH codes are an assembly of models and correlations for simulation of physical phenomena and behavior of system parameters in temporal domain. In some cases, there are alternative sub-models, or several different correlations for calculation of a specific phenomenon of interest. There are also “user options” for choosing one of several models or correlations in performing a specific code computation. Dynamic characteristics of TH calculations add more complexity to the code calculation, meaning for example, that specific code models and correlations invoked are sequence-dependent, and based certain (dynamic) conditions being satisfied. Structural uncertainty assessment (model uncertainty) for a single model will be discussed by considering “correction factor”, “bias”, and also through Bayesian sub-model output updating with available experimental evidence. In case of multiple alternative models, several techniques including dynamic model switching, user controlled model selection, model mixing, will be discussed. This paper discusses the challenges in treatment of the structural uncertainties in Thermal-Hydraulics system codes. Subjectivity and dependency on expert judgment in some of the solutions leaves some concerns on context of such systematic solutions to utilize imperfect and partially relevant data and information.

The last few decades have seen considerable progress in the quantification of environmental model uncertainty. Initially the emphasis has been on uncertainty in model parameters. A more recent trend has been to consider uncertainties in both model structure and parameters, most commonly by analyzing jointly predictions generated by several alternative models of the environment. This has been motivated by a growing recognition that the open and complex nature of environmental systems renders them suitable to multiple conceptualizations and mathematical descriptions. Predictions generated by a single model are prone to statistical bias (by reliance on an invalid model) and underestimation of uncertainty (by under-sampling the relevant model space) (Neuman 2003; Neuman and Wierenga 2003). Some multimodel approaches blend or average statistical results generated by a set of alternative models. A common approach to model averaging is to (1) postulate several alternative models for a site, (a) associate each model with a weight or probability, and (c) generate weighted average predictions and statistics of all the models. Ways to accomplish this have varied; some are included in a public-domain code (Multimodel Analysis or MMA by Poeter and Hill 2007) recently reviewed by Ye (2010). This special issue of SERRA focuses on such and other emerging methods of model and parameter uncertainty quantification. The special issue contains seven papers devoted to model averaging. Diks and Vrugt compare model averaging methods that weigh models in different ways, without always requiring that the weights sum up to unity. The methods are applied to two sites and compared in term of their predictive performance measured by out-of-sample root mean squared prediction error. Sain and Furrer estimate weights based on variation and correlation of alternative hierarchical models. They use a Bayesian hierarchical model to estimate correlation between models and the impact of parameter estimation uncertainty on the weights. Ajami and Gu use the Bayesian Model Averaging (BMA) approach of Raftery et al. (2005) to assess uncertainty in a suite of biogeochemical models having various levels of complexity to simulate the fate and transport of nitrate at a field site in California. Their results demonstrate that whereas single models, regardless of their complexity levels, are incapable of representing all active processes at the site, the 95% uncertainty bounds of BMA bracket 90% to 100% of the observations. Tsai use a variance-window (Tsai and Li 2008) version of Maximum Likelihood (ML) BMA (MLBMA; Neuman 2003; Ye et al. 2004) to quantify model uncertainty in managing groundwater within a thick sandy aquifer in Louisiana where saltwater intrusion is of concern. Alternative models are postulated to reflect uncertainty in conceptualizing hydraulic head boundaries and geostatistical parameterization through variogram models. The results M. Ye (&) Department of Scientific Computing, Florida State University, Tallahassee, FL 32306, USA e-mail: mye@fsu.edu

Time-domain coupled loads analysis (CLA) is used to determine the response of a launch vehicle and payload system to transient forces, such as liftoff, engine ignitions and shutdowns, jettison events, and atmospheric flight loads, such as buffet. CLA, using Hurty/Craig-Bampton (HCB) component models, is the accepted method for the establishment of design-level loads for launch systems. However, uncertainty in the component models flows into uncertainty in predicted system results. Uncertainty in the structural responses during launch is a significant concern because small variations in launch vehicle and payload mode shapes and their interactions can result in significant variations in system loads. Uncertainty quantification (UQ) is used to determine statistical bounds on prediction accuracy based on model uncertainty. In this paper uncertainty is treated at the HCB component-model level. In an effort to account for model uncertainties and statistically bound their effect on CLA predictions, this work combines CLA with UQ in a process termed variational coupled loads analysis (VCLA). The modeling of uncertainty using a parametric approach, in which input parameters are represented by random variables, is common, but its major drawback is the resulting uncertainty is limited to the form of the nominal model. Uncertainty in model form is one of the biggest contributors to uncertainty in complex built-up structures. Model-form uncertainty can be represented using a nonparametric approach based on random matrix theory (RMT). In this work, UQ is performed using the hybrid parametric variation (HPV) method, which combines parametric with nonparametric uncertainty at the HCB component model level. The HPV method requires the selection of dispersion values for the HCB fixed-interface (FI) eigenvalues, and the HCB mass and stiffness matrices. The dispersions are based upon component test-analysis modal correlation results. During VCLA, random component models are assembled into an ensemble of random systems using a Monte Carlo (MC) approach. CLA is applied to each of the ensemble members to produce an ensemble of system-level responses for statistical analysis. The proposed methodology is demonstrated through its application to a buffet loads analysis of NASA’s Space Launch System (SLS) during the transonic regime 50 s after liftoff. Core stage (CS) section shears and moments are recovered, and statistics are computed.KeywordsUncertainty quantificationHurty/Craig-BamptonRandom matrixModel formCoupled loads analysis