Model Structural Errors Research Articles

Treatment of input uncertainty in hydrologic modeling: Doing hydrology backward with Markov chain Monte Carlo simulation

There is increasing consensus in the hydrologic literature that an appropriate framework for streamflow forecasting and simulation should include explicit recognition of forcing and parameter and model structural error. This paper presents a novel Markov chain Monte Carlo (MCMC) sampler, entitled differential evolution adaptive Metropolis (DREAM), that is especially designed to efficiently estimate the posterior probability density function of hydrologic model parameters in complex, high‐dimensional sampling problems. This MCMC scheme adaptively updates the scale and orientation of the proposal distribution during sampling and maintains detailed balance and ergodicity. It is then demonstrated how DREAM can be used to analyze forcing data error during watershed model calibration using a five‐parameter rainfall‐runoff model with streamflow data from two different catchments. Explicit treatment of precipitation error during hydrologic model calibration not only results in prediction uncertainty bounds that are more appropriate but also significantly alters the posterior distribution of the watershed model parameters. This has significant implications for regionalization studies. The approach also provides important new ways to estimate areal average watershed precipitation, information that is of utmost importance for testing hydrologic theory, diagnosing structural errors in models, and appropriately benchmarking rainfall measurement devices.

Equifinality of formal (DREAM) and informal (GLUE) Bayesian approaches in hydrologic modeling?

In recent years, a strong debate has emerged in the hydrologic literature regarding what constitutes an appropriate framework for uncertainty estimation. Particularly, there is strong disagreement whether an uncertainty framework should have its roots within a proper statistical (Bayesian) context, or whether such a framework should be based on a different philosophy and implement informal measures and weaker inference to summarize parameter and predictive distributions. In this paper, we compare a formal Bayesian approach using Markov Chain Monte Carlo (MCMC) with generalized likelihood uncertainty estimation (GLUE) for assessing uncertainty in conceptual watershed modeling. Our formal Bayesian approach is implemented using the recently developed differential evolution adaptive metropolis (DREAM) MCMC scheme with a likelihood function that explicitly considers model structural, input and parameter uncertainty. Our results demonstrate that DREAM and GLUE can generate very similar estimates of total streamflow uncertainty. This suggests that formal and informal Bayesian approaches have more common ground than the hydrologic literature and ongoing debate might suggest. The main advantage of formal approaches is, however, that they attempt to disentangle the effect of forcing, parameter and model structural error on total predictive uncertainty. This is key to improving hydrologic theory and to better understand and predict the flow of water through catchments.

So just why would a modeller choose to be incoherent?

This article provides an extended response to the criticisms of the GLUE methodology by Mantovan and Todini [Mantovan, P., Todini, E., 2006. Hydrological forecasting uncertainty assessment: incoherence of the GLUE methodology. J. Hydrol. 330, 368–381]. It is shown that the formal Bayesian identification of models is a special case of GLUE that can be used where the modeller is prepared to make very strong assumptions about the nature of the modelling errors. Under such assumptions, GLUE can be coherent in the sense of Manotvan and Todini. In real applications, however, with multiple sources of uncertainty including model structural error, their strong definition of coherence is shown to be inapplicable to the extent that the choice of a formal likelihood function based on a simple error structure may be an incoherent choice. It is demonstrated by some relatively minor modifications of their hypothetical example that misspecification of the error model and the non-stationarities associated with the presence of input error and model structural error in the Bayes approach will then produce well-defined but incorrect parameter distributions. This empirical result is quite independent of GLUE, but the flexibility of the GLUE approach may then prove to be an advantage in providing more coherent and robust choices of model evaluation in these cases and, by analogy, in other non-ideal cases for real applications. At the current time it is difficult to make a reasoned choice between methods of uncertainty estimation for real applications because of a lack of understanding of the real information content of data in conditioning models.

Evaluating the parameter identifiability and structural validity of a probability-distributed model for soil moisture

The objective of this study is to evaluate the performance of a model that simulates local and spatial average soil moisture in a watershed. The model uses well-known expressions for infiltration, evapotranspiration, and groundwater recharge to describe soil moisture dynamics at the local scale. Then, the spatial mean soil moisture is simulated by integrating the local behavior over a probability density function that characterizes the spatial variability of soil saturation. Ultimately, the model requires input of precipitation and potential evapotranspiration (PET) time series and six parameters to simulate the dynamics of the spatial average soil moisture. The model is applied to the Fort Cobb watershed in Oklahoma using one year of data from September 2005 through August 2006. Model performance is evaluated in three main ways. First, the model’s ability to reproduce observed local and spatial average soil moisture through calibration is examined. Second, the identifiability and stability of the parameter values are considered to assess uncertainty in the parameter values and errors in the model’s mathematical structure. Third, a new method is developed that uses the sensitivities of soil moisture to precipitation and PET to assess the impacts of parameter uncertainty and structural errors on forecasts for unobserved conditions. At the local scale, the model reproduces the soil moisture with a similar degree of accuracy as a more physically-based model (HYDRUS 1D), and both models exhibit some structural errors. The model can also be calibrated to approximately reproduce the spatial average soil moisture observations. However, the model produces a relatively wide range of plausible sensitivities and this range varies depending on the window of time from which the parameters are calibrated. This result implies that parameter uncertainty and model structural errors contribute substantially to model uncertainty for unobserved conditions.

Vertical mixing in atmospheric tracer transport models: error characterization and propagation

Abstract. Imperfect representation of vertical mixing near the surface in atmospheric transport models leads to uncertainties in modelled tracer mixing ratios. When using the atmosphere as an integrator to derive surface-atmosphere exchange from mixing ratio observations made in the atmospheric boundary layer, this uncertainty has to be quantified and taken into account. A comparison between radiosonde-derived mixing heights and mixing heights derived from ECMWF meteorological data during May–June 2005 in Europe revealed random discrepancies of about 40% for the daytime with insignificant bias errors, and much larger values approaching 100% for nocturnal mixing layers with bias errors also exceeding 50%. The Stochastic Time Inverted Lagrangian Transport (STILT) model was used to propagate this uncertainty into CO2 mixing ratio uncertainties, accounting for spatial and temporal error covariance. Average values of 3 ppm were found for the 2 month period, indicating that this represents a large fraction of the overall uncertainty. A pseudo data experiment shows that the error propagation with STILT avoids biases in flux retrievals when applied in inversions. The results indicate that flux inversions employing transport models based on current generation meteorological products have misrepresented an important part of the model error structure likely leading to biases in the estimated mean and uncertainties. We strongly recommend including the solution presented in this work: better, higher resolution atmospheric models, a proper description of correlated random errors, and a modification of the overall sampling strategy.

Corrected score estimation in the proportional hazards model with misclassified discrete covariates.

We consider Cox proportional hazards regression when the covariate vector includes error-prone discrete covariates along with error-free covariates, which may be discrete or continuous. The misclassification in the discrete error-prone covariates is allowed to be of any specified form. Building on the work of Nakamura and his colleagues, we present a corrected score method for this setting. The method can handle all three major study designs (internal validation design, external validation design, and replicate measures design), both functional and structural error models, and time-dependent covariates satisfying a certain 'localized error' condition. We derive the asymptotic properties of the method and indicate how to adjust the covariance matrix of the regression coefficient estimates to account for estimation of the misclassification matrix. We present the results of a finite-sample simulation study under Weibull survival with a single binary covariate having known misclassification rates. The performance of the method described here was similar to that of related methods we have examined in previous works. Specifically, our new estimator performed as well as or, in a few cases, better than the full Weibull maximum likelihood estimator. We also present simulation results for our method for the case where the misclassification probabilities are estimated from an external replicate measures study. Our method generally performed well in these simulations. The new estimator has a broader range of applicability than many other estimators proposed in the literature, including those described in our own earlier work, in that it can handle time-dependent covariates with an arbitrary misclassification structure. We illustrate the method on data from a study of the relationship between dietary calcium intake and distal colon cancer.

Elucidating Model Inadequacies in a Cloud Parameterization by Use of an Ensemble-Based Calibration Framework

AbstractEvery cloud parameterization contains structural model errors. The source of these errors is difficult to pinpoint because cloud parameterizations contain nonlinearities and feedbacks. To elucidate these model inadequacies, this paper uses a general-purpose ensemble parameter estimation technique. In principle, the technique is applicable to any parameterization that contains a number of adjustable coefficients. It optimizes or calibrates parameter values by attempting to match predicted fields to reference datasets. Rather than striving to find the single best set of parameter values, the output is instead an ensemble of parameter sets. This ensemble provides a wealth of information. In particular, it can help uncover model deficiencies and structural errors that might not otherwise be easily revealed. The calibration technique is applied to an existing single-column model (SCM) that parameterizes boundary layer clouds. The SCM is a higher-order turbulence closure model. It is closed using a multivariate probability density function (PDF) that represents subgrid-scale variability. Reference datasets are provided by large-eddy simulations (LES) of a variety of cloudy boundary layers. The calibration technique locates some model errors in the SCM. As a result, empirical modifications are suggested. These modifications are evaluated with independent datasets and found to lead to an overall improvement in the SCM’s performance.

Product-Error-Driven Uncertainty Model for Probabilistic Quantitative Precipitation Estimation with NEXRAD Data

Abstract Although it is broadly acknowledged that the radar-rainfall (RR) estimates based on the U.S. national network of Weather Surveillance Radar-1988 Doppler (WSR-88D) stations contain a high degree of uncertainty, no methods currently exist to inform users about its quantitative characteristics. The most comprehensive characterization of this uncertainty can be achieved by delivering the products in a probabilistic rather than the traditional deterministic form. The authors are developing a methodology for probabilistic quantitative precipitation estimation (PQPE) based on weather radar data. In this study, they present the central element of this methodology: an empirically based error structure model for the RR products. The authors apply a product-error-driven (PED) approach to obtain a realistic uncertainty model. It is based on the analyses of six years of data from the Oklahoma City, Oklahoma, WSR-88D radar (KTLX) processed with the Precipitation Processing System algorithm of the NEXRAD system. The modeled functional-statistical relationship between RR estimates and corresponding true rainfall consists of two components: a systematic distortion function and a stochastic factor quantifying remaining random errors. The two components are identified using a nonparametric functional estimation apparatus. The true rainfall is approximated with rain gauge data from the Oklahoma Mesonet and the U.S. Department of Agriculture (USDA) Agricultural Research Service Micronet networks. The RR uncertainty model presented here accounts for different time scales, synoptic regimes, and distances from the radar. In addition, this study marks the first time in which results on RR error correlation in space and time are presented.

Bayesian uncertainty analysis in distributed hydrologic modeling: A case study in the Thur River basin (Switzerland)

Calibration and uncertainty analysis in hydrologic modeling are affected by measurement errors in input and response and errors in model structure. Recently, extending similar approaches in discrete time, a continuous time autoregressive error model was proposed for statistical inference and uncertainty analysis in hydrologic modeling. The major advantages over discrete time formulation are the use of a continuous time error model for describing continuous processes, the possibility of accounting for seasonal variations of parameters in the error model, the easier treatment of missing data or omitted outliers, and the opportunity for continuous time predictions. The model was developed for the Chaohe Basin in China and had some features specific for this semiarid climatic region (in particular, the seasonal variation of parameters in the error model in response to seasonal variation in precipitation). This paper tests and extends this approach with an application to the Thur River basin in Switzerland, which is subject to completely different climatic conditions. This application corroborates the general applicability of the approach but also demonstrates the necessity of accounting for the heavy tails in the distributions of residuals and innovations. This is done by replacing the normal distribution of the innovations by a Student t distribution, the degrees of freedom of which are adapted to best represent the shape of the empirical distribution of the innovations. We conclude that with this extension, the continuous time autoregressive error model is applicable and flexible for hydrologic modeling under different climatic conditions. The major remaining conceptual disadvantage is that this class of approaches does not lead to a separate identification of model input and model structural errors. The major practical disadvantage is the high computational demand characteristic for all Markov chain Monte Carlo techniques.

Hydrological modelling of the Chaohe Basin in China: Statistical model formulation and Bayesian inference

Calibration of hydrologic models is very difficult because of measurement errors in input and response, errors in model structure, and the large number of non-identifiable parameters of distributed models. The difficulties even increase in arid regions with high seasonal variation of precipitation, where the modelled residuals often exhibit high heteroscedasticity and autocorrelation. On the other hand, support of water management by hydrologic models is important in arid regions, particularly if there is increasing water demand due to urbanization. The use and assessment of model results for this purpose require a careful calibration and uncertainty analysis. Extending earlier work in this field, we developed a procedure to overcome (i) the problem of non-identifiability of distributed parameters by introducing aggregate parameters and using Bayesian inference, (ii) the problem of heteroscedasticity of errors by combining a Box–Cox transformation of results and data with seasonally dependent error variances, (iii) the problems of autocorrelated errors, missing data and outlier omission with a continuous-time autoregressive error model, and (iv) the problem of the seasonal variation of error correlations with seasonally dependent characteristic correlation times. The technique was tested with the calibration of the hydrologic sub-model of the Soil and Water Assessment Tool (SWAT) in the Chaohe Basin in North China. The results demonstrated the good performance of this approach to uncertainty analysis, particularly with respect to the fulfilment of statistical assumptions of the error model. A comparison with an independent error model and with error models that only considered a subset of the suggested techniques clearly showed the superiority of the approach based on all the features (i)–(iv) mentioned above.

The 15-Country Collaborative Study of Cancer Risk among Radiation Workers in the Nuclear Industry: Study of Errors in Dosimetry

To provide direct estimates of cancer risk after low-dose protracted exposure to ionizing radiation, a large-scale epidemiological study of nuclear industry workers was conducted in 15 countries. As part of this study, identification and quantification of errors in historical recorded doses was conducted based on a review of dosimetric practices and technologies in participating facilities. The main sources of errors on doses from "high-energy" photons (100-3000 keV) were identified as the response of dosimeters in workplace exposure conditions and historical calibration practices. Errors related to dosimetry technology and radiation fields were quantified to derive period- and facility-specific estimates of bias and uncertainties in recorded doses. This was based on (1) an evaluation of predominant workplace radiation from measurement studies and dosimetry expert assessment and (2) an estimation of the energy and geometry response of dosimeters used historically in study facilities. Coefficients were derived to convert recorded doses to H(p) (10) and organ dose, taking into account different aspects of the calibration procedures. A parametric, lognormal error structure model was developed to describe errors in doses as a function of facility and time period. Doses from other radiation types, particularly neutrons and radionuclide intake, could not be adequately reconstructed in the framework of the 15-Country Study. Workers with substantial doses from these radiation types were therefore identified and excluded from analyses. Doses from "lower-energy" photons (<100 keV) and from "higher-energy" photons (>3 MeV) were estimated to be small.

Comment on “Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology” by Pietro Mantovan and Ezio Todini

This comment is a response to the criticisms of the GLUE methodology by [Mantovan, P., Todini, E., 2006. Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology, J. Hydrology, 2006]. In this comment it is shown that the formal Bayesian identification of models is a special case of GLUE that can be used where the modeller is prepared to make very strong assumptions about the nature of the modelling errors. For the hypothetical study of Mantovan and Todini, exact assumptions were assumed known for the formal Bayesian identification, but were then ignored in the application of GLUE to the same data. We show that a more reasonable application of GLUE to this problem using similar prior knowledge shows that gives equally coherent results to the formal Bayes identification. In real applications, subject to input and model structural error it is suggested that the coherency condition of MT06 cannot hold at the single observation level and that the choice of a formal Bayesian likelihood function may then be incoherent. In these (more interesting) cases, GLUE can be coherent in the application of likelihood measures based on blocks of data, but different choices of measures and blocks effectively represent different beliefs about the information content of data in real applications with input and model structural errors.

Evolutionary computation-based approach for model error correction and calibration

Calibration is typically used for improving the predictability of mechanistic simulation models by adjusting a set of model parameters and fitting model predictions to observations. Calibration does not, however, account for or correct potential misspecifications in the model structure, limiting the accuracy of modeled predictions. This paper presents a new approach that addresses both parameter error and model structural error to improve the predictive capabilities of a model. The new approach simultaneously conducts a numeric search for model parameter estimation and a symbolic (regression) search to determine a function to correct misspecifications in model equations. It is based on an evolutionary computation approach that integrates genetic algorithm and genetic programming operators. While this new approach is designed generically and can be applied to a broad array of mechanistic models, it is demonstrated for an illustrative case study involving water quality modeling and prediction. Results based on extensive testing and evaluation, show that the new procedure performs consistently well in fitting a set of training data as well as predicting a set of validation data, and outperforms a calibration procedure and an empirical model fitting procedure.

Evaluation of Model Complexity and Input Uncertainty of Field‐Scale Water Flow and Salt Transport

Prediction of large‐scale vadose zone water flow and salt transport is affected by errors due to uncertainties in model structure, that is, complexity of representation of hydrologic processes and model input uncertainties such as parameter values, boundary conditions, and initial conditions. Selection of an appropriate level of model complexity must consider these two sources of uncertainty. We illustrate this selection process for the prediction of field‐scale crop transpiration and salt drainage in irrigated agriculture given a limited number of point‐scale measurements. Various levels of model input uncertainty in hydraulic properties and infiltration rates are considered, representative of a range of spatial heterogeneities. Model complexity is defined relative to a “true” model, in terms of the level of spatial and temporal averaging used in the approximate model. This set‐up allows for the separation of the total model error into different terms representing the model input error and the structural model error. Results show that the relative contribution of structural model error to the total model error decreases as spatial heterogeneity or uncertainty in the model input increases. Model input uncertainty may be reduced by taking a larger number of point‐scale measurements. Our analysis further illustrates that there exists an optimal trade‐off between model complexity and model input uncertainty. It suggests that complex models may be replaced by more simple ones as long as the resulting structural model error is smaller than the prediction errors due to input uncertainty. The methodology provides a straightforward and objective approach to identifying an optimal level of model complexity as a function of the degree of field‐scale heterogeneity and data availability.

Influence of errors in radar rainfall estimates on hydrological modeling prediction uncertainty

This study aims to assess the impact of a class of radar rainfall errors on prediction uncertainty of a conceptual water balance model. Uncertainty assessment is carried out by means of the Generalized Likelihood Uncertainty Estimation procedure (GLUE). The effects of model input and structural error are separated, and the potential for compensating errors between them is investigated. It is shown that the radar rainfall bias term operates in a multiplicative sense on the model structural uncertainty, by either magnifying or reducing it according to the sign of the bias. The results show also that adjustment of radar rainfall, aimed to remove local biases and to reduce random errors, allows ensuring that a larger percentage of the observed flows are enclosed by the uncertainty bounds, with respect to nonadjusted radar input. However, this is obtained at the price of increasing the wideness of the uncertainty bounds. This effect is emphasized with increasing the radar beam elevation. As a second step, the issue of the impact of radar rainfall estimation error on model parameter distribution and parameter transferability across sites under the radar umbrella is examined. Radar data at different radar beam elevations are used to simulate radar estimation errors at different distances from the radar site and to analyze the impact of these errors on prediction uncertainty. The results show that distortion of parameter distribution due to radar error may be considerable and that adjustment of radar rainfall estimates improves the regionalization potential of radar‐based precipitation estimates (at least for ranges less than 70 km).

Transient response, model structure and systematic errors in hybrid respirometers: structural identifiability analysis based on OUR and DO measurements

Experimental respirometry shows oxygen uptake rate (OUR) transients which are not predicted by standard ASM models. These transients were analysed from a detailed modelling of hybrid respirometers. Structural identifiability analysis of several models based on local isomorphism transformation approach is carried out. It can be concluded that the knowledge of the OUR does not help positively on an accurate determination of biokinetic parameters. The measure of dissolved oxygen (DO) is just enough to calculate the group of identifiable parameters from the models in a calibration process. The analysis suggests to use only the DO data to avoid the systematic errors associated to OUR calculations. Furthermore, the existence of OUR transients after a precise modelling of the hybrid respirometer is deduced. The duration of these transients are the result of the coupling between the kinetics of the biological process and the residence time in the respiration chamber of the instrument. An expression for the duration of these transients is proposed.

Errors and uncertainty in physically-based rainfall-runoff modelling of catchment change effects

The error in physically-based rainfall-runoff modelling is broken into components, and these components are assigned to three groups: (1) model structure error, associated with the model’s equations; (2) parameter error, associated with the parameter values used in the equations; and (3) run time error, associated with rainfall and other forcing data. The error components all contribute to “integrated” errors, such as the difference between simulated and observed runoff, but their individual contributions cannot usually be isolated because the modelling process is complex and there is a lack of knowledge about the catchment and its hydrological responses . A simple model of the Slapton Wood Catchment is developed within a theoretical framework in which the catchment and its responses are assumed to be known perfectly. This makes it possible to analyse the contributions of the error components when predicting the effects of a physical change in the catchment. The standard approach to predicting change effects involves: (1) running “unchanged” simulations using current parameter sets; (2) making adjustments to the sets to allow for physical change; and (3) running “changed” simulations. Calibration or uncertainty-handling methods such as GLUE are used to obtain the current sets based on forcing and runoff data for a calibration period, by minimising or creating statistical bounds for the “integrated” errors in simulations of runoff. It is shown that current parameter sets derived in this fashion are unreliable for predicting change effects, because of model structure error and its interaction with parameter error, so caution is needed if the standard approach is to be used when making management decisions about change in catchments.

Real-Time Data Assimilation for Operational Ensemble Streamflow Forecasting

Abstract Operational flood forecasting requires that accurate estimates of the uncertainty associated with model-generated streamflow forecasts be provided along with the probable flow levels. This paper demonstrates a stochastic ensemble implementation of the Sacramento model used routinely by the National Weather Service for deterministic streamflow forecasting. The approach, the simultaneous optimization and data assimilation method (SODA), uses an ensemble Kalman filter (EnKF) for recursive state estimation allowing for treatment of streamflow data error, model structural error, and parameter uncertainty, while enabling implementation of the Sacramento model without major modification to its current structural form. Model parameters are estimated in batch using the shuffled complex evolution metropolis stochastic-ensemble optimization approach (SCEM-UA). The SODA approach was implemented using parallel computing to handle the increased computational requirements. Studies using data from the Leaf River, Mississippi, indicate that forecast performance improvements on the order of 30% to 50% can be realized even with a suboptimal implementation of the filter. Further, the SODA parameter estimates appear to be less biased, which may increase the prospects for finding useful regionalization relationships.

A BAYESIAN APPROACH TO RETRANSFORMATION BIAS IN TRANSFORMED REGRESSION

Ecological data analysis often involves fitting linear or nonlinear equations to data after transforming either the response variable, the right side of the equation, or both, so that the standard suite of regression assumptions are more closely met. However, inference is usually done in the natural metric and it is well known that retransforming back to the original metric provides a biased estimator for the mean of the response variable. For the normal linear model, fit under a log-transformation, correction factors are available to reduce this bias, but these factors may not be generally applicable to all model forms or other transformations. We demonstrate that this problem is handled in a straightforward manner using a Bayesian approach, which is general for linear and nonlinear models and other transformations and model error structures. The Bayesian framework provides a predictive distribution for the response variable so that inference can be made at the mean, or over the entire distribution to incorporate the predictive uncertainty.

Unraveling uncertainties in hydrologic model calibration: Addressing the problem of compensatory parameters

In a previous paper, Vrugt et al. (2005) presented a combined parameter and state estimation method, entitled SODA, to improve the treatment of input, output, parameter and model structural error during model calibration. The argument for using SODA is that explicit treatment of all sources of uncertainty should result in parameter estimates that closer represent system properties, instead of parameter values that are compensating for input, output and model structural errors. In this study we provide further support for this claim by applying the SODA method to the calibration of a simple 2‐parameter snow model, using data from the Lake Eldora SNOwpack TELemetry (SNOTEL) site in Colorado, USA.