Sources Of Model Error Research Articles

A study of the impact of parameter optimization on ENSO predictability with an intermediate coupled model

Model error is a major obstacle for enhancing the forecast skill of El Nino-Southern Oscillation (ENSO). Among three kinds of model error sources—dynamical core misfitting, physical scheme approximation and model parameter errors, the model parameter errors are treatable by observations. Based on the Zebiak-Cane model, an ensemble coupled data assimilation system is established to study the impact of parameter optimization (PO) on ENSO predictions within a biased twin experiment framework. “Observations” of sea surface temperature anomalies drawn from the “truth” model are assimilated into a biased prediction model in which model parameters are erroneously set from the “truth” values. The degree by which the assimilation and prediction with or without PO recover the “truth” is a measure of the impact of PO. Results show that PO improves ENSO predictability—enhancing the seasonal-interannual forecast skill by about 18 %, extending the valid lead time up to 33 % and ameliorating the spring predictability barrier. Although derived from idealized twin experiments, results here provide some insights when a coupled general circulation model is initialized from the observing system.

Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems

This paper proposes a Bayesian methodology to integrate model verification, validation, and calibration activities for the purpose of overall uncertainty quantification in different types of engineering systems. The methodology is first developed for single-level models, and then extended to systems that are studied using multi-level models that interact with each other. Two types of interactions amongst multi-level models are considered: (1) Type-I, where the output of a lower-level model (component and/or subsystem) becomes an input to a higher level system model, and (2) Type-II, where parameters of the system model are inferred using lower-level models and tests (that describe simplified components and/or isolated physics). The various models, their inputs, parameters, and outputs, experimental data, and various sources of model error are connected through a Bayesian network. The results of calibration, verification, and validation with respect to each individual model are integrated using the principles of conditional probability and total probability, and propagated through the Bayesian network in order to quantify the overall system-level prediction uncertainty. The proposed methodology is illustrated with numerical examples that deal with heat conduction and structural dynamics.

ERROR CONTROLLED USE OF THE TAYLOR ASSUMPTION IN ADAPTIVE HIERARCHICAL MODELING OF DSS

A strategy for macroscale modeling adaptivity in fully nested two-scale computational (first-order) homogenization based on assumed scale separation is proposed. The representative volume element (RVE) for a substructure pertinent to duplex stainless steel is considered with its typical phase morphology, whereby crystal plasticity with hardening is adopted for the subscale material modeling. The quality of the macroscale constitutive response depends on, among the various assumptions regarding the modeling and discretization, the choice of a prolongation condition defining the deformation mapping from the macro- to the subscale This is the sole source of model error discussed in the present contribution. Two common choices are (in hierarchical order) (1) a simplified model based on homogeneous (macroscale) deformation within the RVE, that is the Taylor assumption, and (2) a model employing Dirichlet boundary conditions on the RVE, which is taken as the exact model in the present context. These errors are assessed via computation of the pertinent dual problem. The results show that both the location and the number of qudrature points where the reference model is employed depend on the chosen goal function.

A Novel Affine Arithmetic Method to Solve Optimal Power Flow Problems With Uncertainties

An affine arithmetic (AA) method is proposed in this paper to solve the optimal power flow (OPF) problem with uncertain generation sources. In the AA-based OPF problem, all the state and control variables are treated in affine form, comprising a center value and the corresponding noise magnitudes, to represent forecast, model error, and other sources of uncertainty without the need to assume a probability density function (pdf). The proposed AA-based OPF problem is used to determine the operating margins of the thermal generators in systems with uncertain wind and solar generation dispatch. The AA-based approach is benchmarked against Monte Carlo simulation (MCS) intervals in order to determine its effectiveness. The proposed technique is tested and demonstrated on the IEEE 30-bus system and also a real 1211-bus European system.

Representing model error in ensemble data assimilation

Abstract. The paper investigates a method to represent model error in the ensemble data assimilation (EDA) system. The ECMWF operational EDA simulates the effect of both observations and model uncertainties. Observation errors are represented by perturbations with statistics characterized by the observation error covariance matrix whilst the model uncertainties are represented by stochastic perturbations added to the physical tendencies to simulate the effect of random errors in the physical parameterizations (ST-method). In this work an alternative method (XB-method) is proposed to simulate model uncertainties by adding perturbations to the model background field. In this way the error represented is not just restricted to model error in the usual sense but potentially extends to any form of background error. The perturbations have the same correlation as the background error covariance matrix and their magnitude is computed from comparing the high-resolution operational innovation variances with the ensemble variances when the ensemble is obtained by perturbing only the observations (OBS-method). The XB-method has been designed to represent the short range model error relevant for the data assimilation window. Spread diagnostic shows that the XB-method generates a larger spread than the ST-method that is operationally used at ECMWF, in particular in the extratropics. Three-dimensional normal-mode diagnostics indicate that XB-EDA spread projects more than the spread from the other EDAs onto the easterly inertia-gravity modes associated with equatorial Kelvin waves, tropical dynamics and, in general, model error sources. The background error statistics from the above described EDAs have been employed in the assimilation system. The assimilation system performance showed that the XB-method background error statistics increase the observation influence in the analysis process. The other EDA background error statistics, when inflated by a global factor, generate analyses with 30–50% smaller degree of freedom of signal. XB-EDA background error variances have not been inflated. The presented EDAs have been used to generate the initial perturbations of the ECMWF ensemble prediction system (EPS) of which the XB-EDA induces the largest EPS spread, also in the medium range, leading to a more reliable ensemble. Compared to ST-EDA, XB-EDA leads to a small improvement of the EPS ignorance skill score at day 3 and 7.

CRESP: Towards Optimal Resource Provisioning for MapReduce Computing in Public Clouds

Running MapReduce programs in the cloud introduces this unique problem: how to optimize resource provisioning to minimize the monetary cost or job finish time for a specific job? We study the whole process of MapReduce processing and build up a cost function that explicitly models the relationship among the time cost, the amount of input data, the available system resources (Map and Reduce slots), and the complexity of the Reduce function for the target MapReduce job. The model parameters can be learned from test runs. Based on this cost function, we can solve a number of decision problems, such as the optimal amount of resources that can minimize monetary cost within a job finish deadline, minimize time cost under a certain monetary budget, or find the optimal tradeoffs between time and monetary costs. Experimental results show that the proposed approach performs well on a number of sample MapReduce programs in both the in-house cluster and Amazon EC2. We also conducted a variance analysis on different components of the MapReduce workflow to show the possible sources of modeling error. Our optimization results show that with the proposed approach we can save a significant amount of time and money, compared to randomly selected settings.

An adaptive hybrid EnKF-OI scheme for efficient state-parameter estimation of reactive contaminant transport models

Reactive contaminant transport models are used by hydrologists to simulate and study the migration and fate of industrial waste in subsurface aquifers. Accurate transport modeling of such waste requires clear understanding of the system’s parameters, such as sorption and biodegradation. In this study, we present an efficient sequential data assimilation scheme that computes accurate estimates of aquifer contamination and spatially variable sorption coefficients. This assimilation scheme is based on a hybrid formulation of the ensemble Kalman filter (EnKF) and optimal interpolation (OI) in which solute concentration measurements are assimilated via a recursive dual estimation of sorption coefficients and contaminant state variables. This hybrid EnKF-OI scheme is used to mitigate background covariance limitations due to ensemble under-sampling and neglected model errors. Numerical experiments are conducted with a two-dimensional synthetic aquifer in which cobalt-60, a radioactive contaminant, is leached in a saturated heterogeneous clayey sandstone zone. Assimilation experiments are investigated under different settings and sources of model and observational errors. Simulation results demonstrate that the proposed hybrid EnKF-OI scheme successfully recovers both the contaminant and the sorption rate and reduces their uncertainties. Sensitivity analyses also suggest that the adaptive hybrid scheme remains effective with small ensembles, allowing to reduce the ensemble size by up to 80% with respect to the standard EnKF scheme.

Integrating climate and local factors for geomorphological distribution models

ABSTRACTEarth surface processes (ESPs) drive landscape development and ecosystem processes in high‐latitude regions by creating spatially heterogeneous abiotic and biotic conditions. Ongoing global change may potentially alter the activity of ESPs through feedback on ground conditions, vegetation and the carbon cycle. Consequently, accurate modeling of ESPs is important for improving understanding of the current and future distributions of these processes. The aims of this study were to: (1) integrate climate and multiple local predictors to develop realistic ensemble models for the four key ESPs occurring at high latitudes (slope processes, cryoturbation, nivation and palsa mires) based on the outputs of 10 modern statistical techniques; (2) test whether models of ESPs are improved by incorporating topography, soil and vegetation predictors to climate‐only models; (3) examine the relative importance of these variables in a multivariate setting. Overall, the models showed high transferability with the mean area under curve of a receiver operating characteristics (AUC) ranging from 0.83 to 0.96 and true skill statistics (TSS) from 0.52 to 0.87 for the most complex models. Even though the analyses highlighted the importance of the climate variables as the most influential predictors, three out of four models benefitted from the inclusion of local predictors. We conclude that disregarding local topography and soil conditions in spatial models of ESPs may cause a significant source of error in geomorphological distribution models. Copyright © 2014 John Wiley & Sons, Ltd.

A practical method for optical dispersion model selection and parameters variations in scatterometry analysis with variable n&k's

Optical critical dimension (OCD) scatterometry in recent years became a well-accepted and powerful technique to determine the properties and grating profiles of 2D and 3D microelectronic structures (critical dimensions, side-wall angles, grating and underlying thin film thicknesses) in modern semiconductor manufacturing. However, the optical scatterometry, as any model-based metrology technique, relies on the accuracy of the OCD model which highly depends on the optical properties (the n&k's) of the materials in the structure. In practice, even small deviations in material's optical properties (from nominal model inputs) due to process condition variations might significantly affect the scatterometry measurements. A logical way to deal with this problem is to allow some degree of the n&k's variability of the most affected layer(s) in the OCD model describable by relevant dispersion model(s) (floating n&k's). Essentially, one of the largest complications for the end users (process engineers) is to decide which optical dispersion model (Cauchy, Lorentz, Tauc–Lorentz, etc.) needs to be selected to describe a material under production conditions and which parameters are more sensitive and need to be floated in the OCD model. We developed an approach which, we believe, will result in more straightforward and fast development of the OCD models. This approach provides a possibility to automatically select a most proper dispersion model which has been always an ambiguous decision for most of the end users. This methodology will allow determination of most influential model parameters to vary, eliminating potential sources of modeling error at the initial steps of the OCD modeling. A few examples to illustrate the key ideas and practical use of our procedure have been provided.

High-Altitude Gas Balloon Trajectory Prediction: A Monte Carlo Model

The high-altitude gas balloon is an indispensable tool in atmospheric science, meteorology, and other applications requiring stratospheric observations. A prerequisite of the effectiveness of many types of balloon operations is an accurate trajectory forecasting capability. In particular, targeted flights, sample return missions, or flights of expensive instruments (whose recovery is essential) rely on such models. In this paper, the authors describe a new balloon flight simulation model, which takes into account a range of environmental, physical, and operational uncertainties to generate a predicted trajectory equipped with landing site location error estimates. A key source of error in such models is the incomplete understanding of the drag opposing the rise of balloons in the free atmosphere; here the authors propose a new stochastic drag model based on empirical data derived from thousands of radiosonde flights. Other sources of prediction error are also examined, affecting the accuracy of the flight-path forecast, such as uncertainties in the wind profile and balloon envelope manufacturing variability. The authors show how integrating these elements into a process that generates Monte Carlo ensembles of simulated trajectories has yielded a practically useful flight planning tool, which we have made available to the ballooning community as a free online service.

A systematic approach to identify the sources of tropical SST errors in coupled models using the adjustment of initialised experiments

Understanding the sources of systematic errors in climate models is challenging because of coupled feedbacks and errors compensation. The developing seamless approach proposes that the identification and the correction of short term climate model errors have the potential to improve the modeled climate on longer time scales. In previous studies, initialised atmospheric simulations of a few days have been used to compare fast physics processes (convection, cloud processes) among models. The present study explores how initialised seasonal to decadal hindcasts (re-forecasts) relate transient week-to-month errors of the ocean and atmospheric components to the coupled model long-term pervasive SST errors. A protocol is designed to attribute the SST biases to the source processes. It includes five steps: (1) identify and describe biases in a coupled stabilized simulation, (2) determine the time scale of the advent of the bias and its propagation, (3) find the geographical origin of the bias, (4) evaluate the degree of coupling in the development of the bias, (5) find the field responsible for the bias. This strategy has been implemented with a set of experiments based on the initial adjustment of initialised simulations and exploring various degrees of coupling. In particular, hindcasts give the time scale of biases advent, regionally restored experiments show the geographical origin and ocean-only simulations isolate the field responsible for the bias and evaluate the degree of coupling in the bias development. This strategy is applied to four prominent SST biases of the IPSLCM5A-LR coupled model in the tropical Pacific, that are largely shared by other coupled models, including the Southeast Pacific warm bias and the equatorial cold tongue bias. Using the proposed protocol, we demonstrate that the East Pacific warm bias appears in a few months and is caused by a lack of upwelling due to too weak meridional coastal winds off Peru. The cold equatorial bias, which surprisingly takes 30 years to develop, is the result of an equatorward advection of midlatitude cold SST errors. Despite large development efforts, the current generation of coupled models shows only little improvement. The strategy proposed in this study is a further step to move from the current random ad hoc approach, to a bias-targeted, priority setting, systematic model development approach.

Generalized Residuals for General Models for Contingency Tables With Application to Item Response Theory

Generalized residuals are a tool employed in the analysis of contingency tables to examine possible sources of model error. They have typically been applied to log-linear models and to latent-class models. A general approach to generalized residuals is developed for a very general class of models for contingency tables. To illustrate their use, generalized residuals are applied to models based on item response theory (IRT) models. Such models are commonly applied to analysis of standardized achievement or aptitude tests. To obtain a realistic perspective on application of generalized residuals, actual testing data are employed.

Robust Portfolio Control with Stochastic Factor Dynamics

Portfolio selection is vulnerable to the error-amplifying effects of combining optimization with statistical estimation and model error. For dynamic portfolio control, sources of model error include the evolution of market factors and the influence of these factors on asset returns. We develop portfolio control rules that are robust to this type of uncertainty, applying a stochastic notion of robustness to uncertainty in model dynamics. In this stochastic formulation, robustness reflects uncertainty about the probability law generating market data, and not just uncertainty about model parameters. We analyze both finite- and infinite-horizon problems in a model in which returns are driven by factors that evolve stochastically. The model incorporates transaction costs and leads to simple and tractable optimal robust controls for multiple assets. We illustrate the performance of the controls on historical data. As one would expect, in-sample tests show no evidence of improved performance through robustness—evaluating performance on the same data used to estimate a model leaves no room to capture model uncertainty. However, robustness does improve performance in out-of-sample tests in which the model is estimated on a rolling window of data and then applied over a subsequent time period. By acknowledging uncertainty in the estimated model, the robust rules lead to less aggressive trading and are less sensitive to sharp moves in underlying prices.

Limitations on the accuracy of model predictions of wildland fire behaviour: A state-of-the-knowledge overview

The degree of accuracy in model predictions of wildland fire behaviour characteristics are dependent on the model's applicability to a given situation, the validity of the model's relationships, and the reliability of the model input data. While much progress has been made by fire behaviour research in the past 35 years or so in addressing these three sources of model error, the accuracy in model predictions are still very much at the mercy of our present understanding of the natural phenomena exhibited by free-burning wildland fires and the inherent temporal and spatial variability in the fire environment. This paper will serve as a state-of-the-art primer on the subject of error sources in model predictions of wildland fire behaviour and includes a short historical overview of wildland fire behaviour research as it relates to model development.

Evaluating building energy model performance of LEED buildings: Identifying potential sources of error through aggregate analysis

This paper presents a proposed framework for evaluating building energy model performance of LEED buildings through an aggregate analysis of comparing predicted to actual building energy consumption as well as evaluating how well the building energy model reflects the actual physical building. In this study, we seek to identify potential sources of energy model error by applying the framework to two, identically constructed and co-located, LEED-certified military dormitories. The simulated and actual building energy data is segregated and analyzed by energy type, heating/cooling season, end use, and weekday/weekend to determine the relative extent to which these differences contribute to the total annual difference between predicted and actual consumption as well as examine the model's assumptions regarding occupancy. Overall, the model over-predicted energy consumption in Buildings A and B by 14% and 25%, respectively, as based on comparison between coinciding modeled and actual cooling and heating seasons. The larger model errors appeared in over-estimating natural gas consumption, particularly the heating season's boiler usage. Examining the buildings’ daily water consumption data exhibited significant changes throughout the analysis period, suggesting similar changes in the buildings’ actual occupancy which deviates from the model's assumption of 100% occupancy throughout the analysis period.

Air quality modelling using the Met Office Unified Model (AQUM OS24-26): model description and initial evaluation

Abstract. The on-line air quality model AQUM (Air Quality in the Unified Model) is a limited-area forecast configuration of the Met Office Unified Model which uses the UKCA (UK Chemistry and Aerosols) sub-model. AQUM has been developed with two aims: as an operational system to deliver regional air quality forecasts and as a modelling system to conduct air quality studies to inform policy decisions on emissions controls. This paper presents a description of the model and the methods used to evaluate the performance of the forecast system against the automated UK surface network of air quality monitors. Results are presented of evaluation studies conducted for a year-long period of operational forecast trials and several past cases of poor air quality episodes. The results demonstrate that AQUM tends to over-predict ozone (~8 μg m−3 mean bias for the year-long forecast), but has a good level of responsiveness to elevated ozone episode conditions – a characteristic which is essential for forecasting poor air quality episodes. AQUM is shown to have a negative bias for PM10, while for PM2.5 the negative bias is much smaller in magnitude. An analysis of speciated PM2.5 data during an episode of elevated particulate matter (PM) suggests that the PM bias occurs mainly in the coarse component. The sensitivity of model predictions to lateral boundary conditions (LBCs) has been assessed by using LBCs from two different global reanalyses and by comparing the standard, single-nested configuration with a configuration having an intermediate European nest. We conclude that, even with a much larger regional domain, the LBCs remain an important source of model error for relatively long-lived pollutants such as ozone. To place the model performance in context we compare AQUM ozone forecasts with those of another forecasting system, the MACC (Monitoring Atmospheric Composition and Climate) ensemble, for a 5-month period. An analysis of the variation of model skill with forecast lead time is presented and the insights this provides to the relative sources of error in air quality modelling are discussed.

Large‐scale hydrologic and hydrodynamic modeling of the Amazon River basin

Key Points Remote sensing validation of a hydrologic‐hydrodynamic model of the Amazon Uncertainty of precipitation and river‐floodplain parameters cause model errors Importance of floodplains and backwater effects on flood waves traveling

A global evaluation of fronts and precipitation in the ACCESS model

How global precipitation will change in the future is of great socio-economic importance. It is therefore vital that climate models are able to adequately simulate the characteristics of precipitation and the individual precipitation events. Fronts play an important role in providing precipitation and they can be associated with heavy rain and flooding. In this paper the ACCESS1.3 atmosphere model is evaluated in terms of frontal and non-frontal precipitation. An objective front identification method is applied to data from reanalysis and the model. The fronts are then linked to daily precipitation from observational estimates and the model. The proportion of precipitation associated with fronts and the average intensity of precipitation associated with fronts are then calculated and compared. The frequency of fronts and the proportion of precipitation associated with fronts are well captured by the model. The intensity of precipitation when a front is present is underestimated by the model in most regions, consistent with many previous studies. Decomposing the total precipitation error into components associated with the frequency and intensity of both frontal and non-frontal precipitation shows that the non-frontal precipitation errors contribute the most to the total error, and that there are compensating errors in the model. This regimeand process-based method of model evaluation provides a useful tool to gain deeper understanding into the sources of precipitation errors in climate models.

Analysis and Modeling of Error Sources for Spiral Bevel Gear Honing Machine Based on Multi-Body System Theory

Analyzing some main errors of NC spiral bevel gear honing machine-tool, the elements of geometric error, thermal error and force error etc., which are constructed 39 error sources of the machine by stack method. Based on multi-body system theory, using homogeneous transfer matrix(HTM) method to present six feature matrixes that includes ideal-static matrix, ideal-kinematic matrix, ideal-static translation error matrix, ideal-static rotary error matrix, non-ideal rotary motion error matrix and non-ideal translation motion error matrix, which are built between of adjacent body. Finally, the synthetic volumetric error model of six-axis honing machine-tool is established.

Inherent Errors in Pollutant Build‐Up Estimation in Considering Urban Land Use as a Lumped Parameter

Stormwater quality modeling results are subject to uncertainty. The variability of input parameters is an important source of overall model error. An in-depth understanding of the variability associated with input parameters can provide knowledge on the uncertainty associated with these parameters and can assist in uncertainty analysis of stormwater quality models and decision making based on modeling outcomes. This paper discusses the outcomes of a research study undertaken to analyze the variability related to pollutant build-up parameters in stormwater quality modeling. The study was based on the analysis of pollutant build-up samples collected from 12 road surfaces in residential, commercial, and industrial land uses. It was found that build-up characteristics vary appreciably even within the same land use. Therefore, using land use as a lumped parameter would contribute significant uncertainties in stormwater quality modeling. Additionally, it was found that the variability in pollutant build-up can be significant depending on the pollutant type. This underlines the importance of taking into account specific land use characteristics and targeted pollutant species when undertaking uncertainty analysis of stormwater quality models or in interpreting the modeling outcomes.