Model Validation Framework Research Articles

Integrating traditional and non-traditional model risk frameworks in credit scoring

Background: An improved understanding of the reasoning behind model decisions can enhance the use of machine learning (ML) models in credit scoring. Although ML models are widely regarded as highly accurate, the use of these models in settings that require explanation of model decisions has been limited because of the lack of transparency. Especially in the banking sector, model risk frameworks frequently require a significant level of model interpretability.Aim: The aim of the article is to evaluate traditional model risk frameworks to determine their appropriateness when validating ML models in credit scoring and enhance the use of ML models in regulated environments by introducing a ML interpretability technique in model validation frameworks.Setting: The research considers model risk frameworks and regulatory guidelines from various international institutions.Method: The research is qualitative in nature and shows how through integrating traditional and non-traditional model risk frameworks, the practitioner can leverage trusted techniques and extend traditional frameworks to address key principles such as transparency.Results: The article proposes a model risk framework that utilises Shapley values to improve the explainability of ML models in credit scoring. Practical validation tests are proposed to enable transparency of model input variables in the validation process of ML models.Conclusion: Our results show that one can formulate a comprehensive validation process by integrating traditional and non-traditional frameworks.Contribution: This study contributes to existing model risk literature by proposing a new model validation framework that utilises Shapley values to explain ML model predictions in credit scoring.

Blood Lipoproteins Shape the Phenotype and Lipid Content of Early Atherosclerotic Lesion Macrophages: A Dual-Structured Mathematical Model

Macrophages in atherosclerotic lesions exhibit a spectrum of behaviours or phenotypes. The phenotypic distribution of monocyte-derived macrophages (MDMs), its correlation with MDM lipid content, and relation to blood lipoprotein densities are not well understood. Of particular interest is the balance between low density lipoproteins (LDL) and high density lipoproteins (HDL), which carry bad and good cholesterol respectively. To address these issues, we have developed a mathematical model for early atherosclerosis in which the MDM population is structured by phenotype and lipid content. The model admits a simpler, closed subsystem whose analysis shows how lesion composition becomes more pathological as the blood density of LDL increases relative to the HDL capacity. We use asymptotic analysis to derive a power-law relationship between MDM phenotype and lipid content at steady-state. This relationship enables us to understand why, for example, lipid-laden MDMs have a more inflammatory phenotype than lipid-poor MDMs when blood LDL lipid density greatly exceeds HDL capacity. We show further that the MDM phenotype distribution always attains a local maximum, while the lipid content distribution may be unimodal, adopt a quasi-uniform profile or decrease monotonically. Pathological lesions exhibit a local maximum in both the phenotype and lipid content MDM distributions, with the maximum at an inflammatory phenotype and near the lipid content capacity respectively. These results illustrate how macrophage heterogeneity arises in early atherosclerosis and provide a framework for future model validation through comparison with single-cell RNA sequencing data.

Systematic Calibration and Validation Approach for Discrete Element Method (DEM) Modeling of Corn Under Varying Moisture Contents (MC)

Abstract. Discrete Element Method (DEM) based simulation of grain threshing and conveyance processing at relatively dry to wet grain moisture content range has been limited. A comprehensive five-step framework of a DEM grain model calibration and validation of flow and grain impacts on equipment was investigated. The framework was examined by developing DEM models for corn grain moisture conditions sampled during crop harvesting, and simulated hopper discharge corn flow, and corn impact forces on a grain paddle elevator of a crop harvesting machine. Five bulk material response parameters were obtained from two calibration experiments of the angle of repose and direct shear test at the MC levels of 11%, 16%, and 26%. Latin hypercube sampling statistical design of experiment points, Gaussian Process Regression surrogate model, and optimization algorithms were successfully implemented to generate Hertz-Mindlin (HM) with Johnson-Kendall-Roberts (JKR) cohesive model of corn at three moisture content levels. The DEM simulation results predict the corn mass flow rate of hopper discharge with relative errors of -5%, -4%, and 1% for corn at 11%, 16%, and 26% MC levels, respectively. The DEM also predicts corn impact forces on a paddle elevator with a relative error of less than 11%. The established DEM calibration methodology, used for simulation of crop threshing, and harvested grain handling processes at extreme grain moisture content, can accelerate the simulation-based design of crop harvesting equipment. Keywords: Calibration, Corn, Discrete Element Method, Moisture Content, Sensitivity analysis, Validation.

Large-scale spatio-temporal yield estimation via deep learning using satellite and management data fusion in vineyards

Crop yield forecasting is essential for informed farm management decisions. However, most yield forecasting models have low spatial resolution, late-season predictions, and lack validation for unseen years or locations. These limitations likely stem from the scarcity of large-scale, high-resolution yield measurement data collected over multiple years, which is uncommon in commercial specialty crop operations. Additionally, these limitations raise concerns about the models’ utility and generalizability under new environmental and management conditions within or across farms. In this study, we develop a spatio-temporal deep learning model to forecast wine grape yield early in the season, utilizing a large dataset with high spatio-temporal resolution (i.e., yield data from ∼5 million grapevines of eight cultivars observed over four years). The model is trained on weekly 10 m RGB-NIR time-series satellite data from Sentinel 2A-B, fused with categorical and continuous management inputs, including cultivar type, trellis type, row spacing, and canopy spacing. We assess the model’s generalizability by examining its performance on data from unseen years and/or locations and at multiple spatial resolutions. Our results show that combining management data with satellite imagery significantly improves model performance on entirely unseen vineyard blocks at 10 m resolution, achieving an R2 of 0.76, a mean absolute error (MAE) of 4.21 tonnes/hectare, and a mean absolute percent error (MAPE) of 13%. We find that cultivars with considerable year-to-year yield variability tend to exhibit lower predictive performance and may benefit from longer time-series observations for model training to encompass a wide range of environmental and management conditions. We also observe improved estimations from the early season in April to the middle of the growing season in June. In conclusion, the yield forecasting and model validation framework established in this study lays the foundation for training spatio-temporally aware deep learning models on exceptionally large yield datasets with high spatio-temporal resolution. We anticipate that such models will become increasingly prevalent as yield monitors are more frequently deployed in specialty crop operations.

Assessing Serious Spinal Pathology Using Bayesian Network Decision Support: Development and Validation Study.

Identifying and managing serious spinal pathology (SSP) such as cauda equina syndrome or spinal infection in patients presenting with low back pain is challenging. Traditional red flag questioning is increasingly criticized, and previous studies show that many clinicians lack confidence in managing patients presenting with red flags. Improving decision-making and reducing the variability of care for these patients is a key priority for clinicians and researchers. We aimed to improve SSP identification by constructing and validating a decision support tool using a Bayesian network (BN), which is an artificial intelligence technique that combines current evidence and expert knowledge. A modified RAND appropriateness procedure was undertaken with 16 experts over 3 rounds, designed to elicit the variables, structure, and conditional probabilities necessary to build a causal BN. The BN predicts the likelihood of a patient with a particular presentation having an SSP. The second part of this study used an established framework to direct a 4-part validation that included comparison of the BN with consensus statements, practice guidelines, and recent research. Clinical cases were entered into the model and the results were compared with clinical judgment from spinal experts who were not involved in the elicitation. Receiver operating characteristic curves were plotted and area under the curve were calculated for accuracy statistics. The RAND appropriateness procedure elicited a model including 38 variables in 3 domains: risk factors (10 variables), signs and symptoms (17 variables), and judgment factors (11 variables). Clear consensus was found in the risk factors and signs and symptoms for SSP conditions. The 4-part BN validation demonstrated good performance overall and identified areas for further development. Comparison with available clinical literature showed good overall agreement but suggested certain improvements required to, for example, 2 of the 11 judgment factors. Case analysis showed that cauda equina syndrome, space-occupying lesion/cancer, and inflammatory condition identification performed well across the validation domains. Fracture identification performed less well, but the reasons for the erroneous results are well understood. A review of the content by independent spinal experts backed up the issues with the fracture node, but the BN was otherwise deemed acceptable. The RAND appropriateness procedure and validation framework were successfully implemented to develop the BN for SSP. In comparison with other expert-elicited BN studies, this work goes a step further in validating the output before attempting implementation. Using a framework for model validation, the BN showed encouraging validity and has provided avenues for further developing the outputs that demonstrated poor accuracy. This study provides the vital first step of improving our ability to predict outcomes in low back pain by first considering the problem of SSP. RR2-10.2196/21804.

Creating High-Quality Synthetic Health Data: Framework for Model Development and Validation.

Electronic health records are a valuable source of patient information that must be properly deidentified before being shared with researchers. This process requires expertise and time. In addition, synthetic data have considerably reduced the restrictions on the use and sharing of real data, allowing researchers to access it more rapidly with far fewer privacy constraints. Therefore, there has been a growing interest in establishing a method to generate synthetic data that protects patients' privacy while properly reflecting the data. This study aims to develop and validate a model that generates valuable synthetic longitudinal health data while protecting the privacy of the patients whose data are collected. We investigated the best model for generating synthetic health data, with a focus on longitudinal observations. We developed a generative model that relies on the generalized canonical polyadic (GCP) tensor decomposition. This model also involves sampling from a latent factor matrix of GCP decomposition, which contains patient factors, using sequential decision trees, copula, and Hamiltonian Monte Carlo methods. We applied the proposed model to samples from the MIMIC-III (version 1.4) data set. Numerous analyses and experiments were conducted with different data structures and scenarios. We assessed the similarity between our synthetic data and the real data by conducting utility assessments. These assessments evaluate the structure and general patterns present in the data, such as dependency structure, descriptive statistics, and marginal distributions. Regarding privacy disclosure, our model preserves privacy by preventing the direct sharing of patient information and eliminating the one-to-one link between the observed and model tensor records. This was achieved by simulating and modeling a latent factor matrix of GCP decomposition associated with patients. The findings show that our model is a promising method for generating synthetic longitudinal health data that is similar enough to real data. It can preserve the utility and privacy of the original data while also handling various data structures and scenarios. In certain experiments, all simulation methods used in the model produced the same high level of performance. Our model is also capable of addressing the challenge of sampling patients from electronic health records. This means that we can simulate a variety of patients in the synthetic data set, which may differ in number from the patients in the original data. We have presented a generative model for producing synthetic longitudinal health data. The model is formulated by applying the GCP tensor decomposition. We have provided 3 approaches for the synthesis and simulation of a latent factor matrix following the process of factorization. In brief, we have reduced the challenge of synthesizing massive longitudinal health data to synthesizing a nonlongitudinal and significantly smaller data set.

Bayesian uncertainty quantification framework for wake model calibration and validation with historical wind farm power data

AbstractThe expected growth in wind energy capacity requires efficient and accurate models for wind farm layout optimization, control, and annual energy predictions. Although analytical wake models are widely used for these applications, several model components must be better understood to improve their accuracy. To this end, we propose a Bayesian uncertainty quantification framework for physics‐guided data‐driven model enhancement. The framework incorporates turbulence‐related aleatoric uncertainty in historical wind farm data, epistemic uncertainty in the empirical parameters, and systematic uncertainty due to unmodeled physics. We apply the framework to the wake expansion parametrization in the Gaussian wake model and employ historical power data of the Westermost Rough Offshore Wind Farm. We find that the framework successfully distinguishes the three sources of uncertainty in the joint posterior distribution of the parameters. On the one hand, the framework allows for wake model calibration by selecting the maximum a posteriori estimators for the empirical parameters. On the other hand, it facilitates model validation by separating the measurement error and the model error distribution. In addition, the model adequacy and the effect of unmodeled physics are assessable via the posterior parameter uncertainty and correlations. Consequently, we believe that the Bayesian uncertainty quantification framework can be used to calibrate and validate existing and upcoming physics‐guided models.

Validation framework for epidemiological models with application to COVID-19 models.

Mathematical models have been an important tool during the COVID-19 pandemic, for example to predict demand of critical resources such as medical devices, personal protective equipment and diagnostic tests. Many COVID-19 models have been developed. However, there is relatively little information available regarding reliability of model predictions. Here we present a general model validation framework for epidemiological models focused around predictive capability for questions relevant to decision-making end-users. COVID-19 models are typically comprised of multiple releases, and provide predictions for multiple localities, and these characteristics are systematically accounted for in the framework, which is based around a set of validation scores or metrics that quantify model accuracy of specific quantities of interest including: date of peak, magnitude of peak, rate of recovery, and monthly cumulative counts. We applied the framework to retrospectively assess accuracy of death predictions for four COVID-19 models, and accuracy of hospitalization predictions for one COVID-19 model (models for which sufficient data was publicly available). When predicting date of peak deaths, the most accurate model had errors of approximately 15 days or less, for releases 3-6 weeks in advance of the peak. Death peak magnitude relative errors were generally in the 50% range 3-6 weeks before peak. Hospitalization predictions were less accurate than death predictions. All models were highly variable in predictive accuracy across regions. Overall, our framework provides a wealth of information on the predictive accuracy of epidemiological models and could be used in future epidemics to evaluate new models or support existing modeling methodologies, and thereby aid in informed model-based public health decision making. The code for the validation framework is available at https://doi.org/10.5281/zenodo.7102854.

Validation of a Physics-based Prognostic Model with Incomplete Data

While the development of prognostic models is nowadays rather feasible, the implementation and validation thereof can still create many challenges. One of the main challenges is the lack of high-quality input data like operational data, environmental data, maintenance data and the limited amount of degradation or failure data. The uncertainty in the output of the prognostic model needs to be quantified before it can be utilised for either model validation or actual maintenance decision support. This study, therefore, proposes a generic framework for prognostic model validation with limited data based on uncertainty propagation. This is realised by using sensitivity indices, correlation coefficients, Monte Carlo simulations and analytical approaches. For demonstration purposes, a rail wear prognostic model is used. The demonstration concludes that by following the generic framework, the prognostic model can be validated, and as a result, realistic maintenance advice can be given to rail infrastructure managers, even when limited data is available.

A framework for feasibility-level validation of high-resolution wave hindcast models

The value of long-term wave hindcasts for investigating wave climates, wave energy resources, and extreme wave conditions has motivated research developing, calibrating and validating wave hindcast models. Past hindcast model validation studies examined the accuracy in modeling bulk wave parameters of overall sea states without considering the dependency of the model's skill within different sea states. In the present study, a framework for wave hindcast model validation is developed by examining the model accuracy for the most frequently occurring sea states, sea states contributing the most energy to total wave power, sea states associated with hurricane events, and those with the largest model error. Validations using bulk wave parameters and frequency-directional spectra at these key sea states and extreme wave conditions based on univariate and bivariate-contour methods provide insights to improve model accuracy, identifying the model's strong and weak points, and pathways for improvement, e.g., modeling wave-current interactions and adjusting wind data. This study adds to a growing body of research demonstrating that a carefully calibrated and verified spectral wave hindcast model can be used to estimate key wave energy parameters over a wide range of wave energy climates, as well as their spatial, temporal, frequency, directional, and probabilistic distributions.

Larger territories reduce mortality risk for chimpanzees, wolves, and agents: Multiple lines of evidence in a model validation framework

Validation of agent-based models is underappreciated in scientific studies even though this process is an important part of ensuring that a model is a reliable research tool. Here we propose modifications to a model validation framework and illustrate this framework with a case study of territorial behavior. In species that defend territories, larger territories provide obvious benefits, such as increased access to food, shelter, and mates. An additional potential benefit is that larger territories could provide protection from intergroup conflict. Considerations from geometry indicate that per capita risk of death from intergroup violence should decrease with increasing territory size, insofar as conflict occurs mainly at the periphery. We tested this inference using computer simulations and data from chimpanzees (Pan troglodytes) and gray wolves (Canis lupus). We designed an agent-based model that allows territories to vary freely in size and shape. We present a framework for model validation and apply it to our agent-based model to substantiate its accuracy. Simulations from the validated model confirmed the predictions from geometry. Under a broad range of parameter values, per capita mortality rate decreased in larger territories. Similarly, using published data on rates of death from intercommunity aggression in 16 chimpanzee communities, as well as new data from 38 wolf packs, we found that per capita mortality rate correlated negatively with a measure of territory size. These findings indicate that in species with lethal intergroup aggression, one simple aspect of territory geometry – size – has strong effects on mortality. Several lines of evidence indicate that the per capita rates of mortality from warfare have decreased in many human societies over time. One factor contributing to this decrease may be the increasing geographic extent of political entities.

Capillary-Driven Ejection of a Droplet from a Micropore into a Channel: A Theoretical Model and a Computational Fluid Dynamics Verification.

In this work, the problem of re-ejection of a permeating droplet through a membrane pore back to the feed channel when the transmembrane pressure (TMP) becomes zero is investigated. This problem is important in the context of oily water filtration using membranes. In particular, in the novel periodic feed pressure technique (PFPT), which has been proposed to combat membrane fouling, the TMP alternates between the operating value and zero in a periodic manner. During the period in which TMP is high, filtration occurs, and when it is zero, cleaning commences. We are particularly interested in what happens to a droplet, initially undergoing permeation, when the TMP becomes zero. It is evident that when the TMP is zero the meniscus inside the pore reverses its motion toward the feed channel rather than toward the permeate side by the action of interfacial tension force. A theoretical model is built to determine the rate at which the meniscus inside the pore advances when the TMP is zero. The conservation of momentum equation is used to establish a one-dimensional model that updates the location of the meniscus with time. The derived model considers both quasi-static and dynamic scenarios. In addition, the model accounts for both the viscosity contrast between the two fluids, as well as the gravity. A computational fluid dynamics (CFD) simulation has been built to provide a framework for model verification and validation. The model, based on quasi-static conditions, provides an overall similar trend to that obtained via CFD analysis. Nevertheless, the quasi-static model predicts a more rapid meniscus advancement inside the pore than the CFD simulation. When the dynamic contact angle is incorporated, very good matching is observed.

Solving the Evidence Interpretability Crisis in Health Technology Assessment: A Role for Mechanistic Models?

Health technology assessment (HTA) aims to be a systematic, transparent, unbiased synthesis of clinical efficacy, safety, and value of medical products (MPs) to help policymakers, payers, clinicians, and industry to make informed decisions. The evidence available for HTA has gaps—impeding timely prediction of the individual long-term effect in real clinical practice. Also, appraisal of an MP needs cross-stakeholder communication and engagement. Both aspects may benefit from extended use of modeling and simulation. Modeling is used in HTA for data-synthesis and health-economic projections. In parallel, regulatory consideration of model informed drug development (MIDD) has brought attention to mechanistic modeling techniques that could in fact be relevant for HTA. The ability to extrapolate and generate personalized predictions renders the mechanistic MIDD approaches suitable to support translation between clinical trial data into real-world evidence. In this perspective, we therefore discuss concrete examples of how mechanistic models could address HTA-related questions. We shed light on different stakeholder's contributions and needs in the appraisal phase and suggest how mechanistic modeling strategies and reporting can contribute to this effort. There are still barriers dissecting the HTA space and the clinical development space with regard to modeling: lack of an adapted model validation framework for decision-making process, inconsistent and unclear support by stakeholders, limited generalizable use cases, and absence of appropriate incentives. To address this challenge, we suggest to intensify the collaboration between competent authorities, drug developers and modelers with the aim to implement mechanistic models central in the evidence generation, synthesis, and appraisal of HTA so that the totality of mechanistic and clinical evidence can be leveraged by all relevant stakeholders.

Framework for Vehicle Dynamics Model Validation

Vehicle dynamics models are widely used in many areas of the automotive industry. The usability of each model depends on how well it is able to mimic the behavior of the real vehicle. Each simulation model must go through a thorough investigation process first, which is called model validation. Although, vehicle dynamics simulation models and methodology for computational model validation are well established fields, to the best of the authors’ knowledge a general framework for vehicle dynamics model validation is still lacking. The research aims to develop a comprehensive methodological framework for vehicle dynamics model validation. In this paper the aim is to present the high level layout of the proposed framework, introducing the main blocks and the tasks related, also addressing some critical issues regarding vehicle dynamics model validation such as validation metrics and vehicle parameter measurement and estimation. An important part of the proposed methodology is a sophisticated vehicle dynamics measurement system, which gives the opportunity to estimate a bunch of vehicle parameters during dynamic testing, which can be useful for several reasons, e.g. fine-tuning the parameters of the Pacejka Magic formula. As a case study some vehicle dynamics test based parameter estimations are shown to justify the raison d’ê and investigate possible applications.

Development of a novel hybrid cognitive model validation framework for implementation under COVID-19 restrictions.

The purpose of this study was to develop a method for validation of cognitive models consistent with the remote working situation arising from COVID‐19 restrictions in place in Spring 2020. We propose a framework for structuring validation tasks and applying a scoring system to determine initial model validity. We infer an objective validity level for cognitive models requiring no in‐person observations, and minimal reliance on remote usability and observational studies. This approach has been derived from the necessity of the COVID‐19 response, however, we believe this approach can lower costs and reduce timelines to initial validation in post‐Covid‐19 studies, enabling faster progress in the development of cognitive engineering systems. A three‐stage hybrid validation framework was developed based on existing validation methods and was adapted to enable compliance with the specific limitations derived from COVID‐19 response restrictions. This validation method includes elements of argument‐based validation combined with a cognitive walkthrough analysis, and reflexivity assessments. We conducted a case study of the proposed framework on a developmental cognitive model of cardiovascular surgery to demonstrate application of a real‐world validation task. This framework can be easily and quickly implemented by a small research team and provides a structured validation method to increase confidence in assumptions as well as to provide evidence to support validity claims in the early stages of model development.

Validation of the Prevention Impacts Simulation Model (PRISM).

IntroductionDemonstrating the validity of a public health simulation model helps to establish confidence in the accuracy and usefulness of a model’s results. In this study we evaluated the validity of the Prevention Impacts Simulation Model (PRISM), a system dynamics model that simulates health, mortality, and economic outcomes for the US population. PRISM primarily simulates outcomes related to cardiovascular disease but also includes outcomes related to other chronic diseases that share risk factors. PRISM is openly available through a web application.MethodsWe applied the model validation framework developed independently by the International Society of Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making modeling task force to validate PRISM. This framework included model review by external experts and quantitative data comparison by the study team.ResultsExternal expert review determined that PRISM is based on up-to-date science. One-way sensitivity analysis showed that no parameter affected results by more than 5%. Comparison with other published models, such as ModelHealth, showed that PRISM produces lower estimates of effects and cost savings. Comparison with surveillance data showed that projected model trends in risk factors and outcomes align closely with secular trends. Four measures did not align with surveillance data, and those were recalibrated.ConclusionPRISM is a useful tool to simulate the potential effects and costs of public health interventions. Results of this validation should help assure health policy leaders that PRISM can help support community health program planning and evaluation efforts.

Development and validation of an Inventory on Teachers’ Beliefs about Action Research (ITBAR) in second language education

ABSTRACT Purpose: Although action research (AR) is considered essential for teacher development in second/foreign language (L2) education, no validated scale has been designed for its measurement. With that in mind, this study attempted to develop and validate an Inventory on Teachers’ Beliefs about Action Research (ITBAR). Design/methodology/approach: For this purpose, a model construction and validation framework consisting of exploratory and confirmatory analyses was used to examine the construct validity of a proposed model. The participants were 287 practicing English language teachers at different institutes, schools, and colleges/universities in Iran. Findings: The results of the exploratory factor analysis indicated that the scale can be best explained by six factors. However, one factor was removed from further analysis because of the low reliability of its components. Thus, confirmatory factor analysis was run with the five remaining factors which, based on the loading of the items and their underlying themes, were named Teacher Empowerment, Practical Issues, Professional Development, Institutional Culture, and Research Engagement. Originality/value: The ITBAR developed in this study is aimed to reflect the realities of L2 classrooms and give direction to the process of critical reflection by providing a context through which teachers articulate their own beliefs about AR.

A model validation framework based on parameter calibration under aleatory and epistemic uncertainty

Model validation methods have been widely used in engineering design to evaluate the accuracy and reliability of simulation models with uncertain inputs. Most of the existing validation methods for aleatory and epistemic uncertainty are based on the Bayesian theorem, which needs a vast number of data to update the posterior distribution of the model parameter. However, when a single simulation is time-consuming, the required simulation cost for the validation of a simulation model may be unaffordable. To overcome this difficulty, a new model validation framework based on parameter calibration under aleatory and epistemic uncertainty is proposed. In the proposed method, a stochastic kriging model is constructed to predict the validity of the candidate simulation model under different uncertainty input parameters. Then, an optimization problem is defined to calibrate the epistemic uncertainty parameters to minimize the discrepancy between the simulation model and the experimental model. K–S test finally decides whether to accept or reject the calibrated simulation model. The performance of the proposed approach is illustrated through a cantilever beam example and a turbine blade validation problem. Results show that the proposed framework can identify the most appropriate parameters to calibrate the simulation model and provide a correct judgment about the validity of the candidate model, which is useful for the validation of simulation models in practical engineering design.

Unified Framework and Survey for Model Verification, Validation and Uncertainty Quantification

Simulation is becoming increasingly important in the development, testing and approval process in many areas of engineering, ranging from finite element models to highly complex cyber-physical systems such as autonomous cars. Simulation must be accompanied by model verification, validation and uncertainty quantification (VV&UQ) activities to assess the inherent errors and uncertainties of each simulation model. However, the VV&UQ methods differ greatly between the application areas. In general, a major challenge is the aggregation of uncertainties from calibration and validation experiments to the actual model predictions under new, untested conditions. This is especially relevant due to high extrapolation uncertainties, if the experimental conditions differ strongly from the prediction conditions, or if the output quantities required for prediction cannot be measured during the experiments. In this paper, both the heterogeneous VV&UQ landscape and the challenge of aggregation will be addressed with a novel modular and unified framework to enable credible decision making based on simulation models. This paper contains a comprehensive survey of over 200 literature sources from many application areas and embeds them into the unified framework. In addition, this paper analyzes and compares the VV&UQ methods and the application areas in order to identify strengths and weaknesses and to derive further research directions. The framework thus combines a variety of VV&UQ methods, so that different engineering areas can benefit from new methods and combinations. Finally, this paper presents a procedure to select a suitable method from the framework for the desired application.

A comprehensive framework for model validation and reliability assessment of container crane structures

A comprehensive framework for model validation and reliability assessment of container crane structures is pursued in this paper. The framework is composed of three phases. In phase I, a parameterized finite element model (FEM) of a typical type of container crane structure at Yangshan deep-water port, Shanghai, is developed for simulating the structure performance at controllable input/parameter settings. Full-scale experiments are conducted at multiple locations of the container crane for the validation assessment of the parameterized FEM. In phase II, high fidelity reliability model (HFRM) of the crane structure is constructed by incorporating the parameterized FEM, failure criteria, and uncertainty from multiple sources. To alleviate the computational burden for running finite element analysis at each function evaluation, surrogate modeling techniques, i.e., the polynomial response surface (PRS) and artificial neural networks (ANN) are applied to build lower fidelity reliability models (LFRM) using simulations generated from the high fidelity reliability model. Quantitative validation metrics, i.e., the area metric and u-pooling metric are applied to assess the degree to which the surrogates represent the high fidelity model in presence of uncertainty. Finally in phase III, the reliability of the container crane structure is assessed by estimating the probability of failure based on the surrogates of the high fidelity reliability model through structure reliability methods. The proposed framework provides a scientific and organized procedure for model validation activities, surrogate model building, and efficient reliability assessment to container cranes and other complex structures in engineering if necessary.