Model Validation Techniques Research Articles

INTEGRATING A SHORT SIMULATION PROJECT INTO AN INTRODUCTORY TRANSPORT PHENOMENA COURSE

Transport phenomena is an integral part of many engineering curricula. Some programs, particularly in chemical engineering, teach transport phenomena in an integrated manner, but most teach heat, mass and momentum transfer in separate courses. Since powerful computers and software packages are now available to solve transport problems numerically, many current transport phenomena courses incorporate these tools for demonstration purposes. Many institutions also offer advanced level courses focusing on the use of this type of software, but the intuitive nature of modern software packages also permits the integration of these tools directly into introductory courses.This paper summarizes efforts made in the Department of Process Engineering and Applied Science at Dalhousie University to integrate one such tool, COMSOL Multiphysics, directly into the introductory transport phenomena course. In an introductory course, the time available to study software tools is inherently limited, and therefore it is necessary to introduce these topics as efficiently as possible. In our course, several hands-on tutorials are used to introduce students to COMSOL. We avoid focusing too much on numerical methods, error estimation techniques, or even detailed model validation techniques. Instead, these tutorials focus on using commercial software to solve transport problems and demonstrating good modeling practices. Near the end of the course, students are asked to complete a short modeling project. In the latest iteration, students were asked to reproduce a simulation in a recent journal publication, to expand on the results, and comment on the significance of their findings. The main challenges related to implementing such a project include selecting problems that do not have unreasonably high computational requirements, the increased workload for teaching assistants, the required level of expertise among teaching assistants, and the increased workload for students.

Near-Infrared Reflectance Spectroscopy as a Method for Determining Organic Carbon Concentrations in Soil

Near-infrared reflectance spectroscopy (NIRS) has the potential to be a reliable method for accurately quantifying soil organic carbon (SOC). The objective of this study was to evaluate NIRS as a method for predicting SOC. Partial least squares (PLS) regression was used to predict SOC from soil reflectance values or the first derivative of the reflectance values. Two model validation techniques were evaluated: One was a full cross-validation and in the other 30 percent of the samples were removed from the calibration data set and then tested using the calibrated model. Significant relationships were observed for predicted SOC when compared to laboratory-measured SOC for all models evaluated, regardless of validation technique. The prediction models using the first derivative of the reflectance values outperformed prediction models using the reflectance values alone. In conclusion, NIRS can be used as a quick and accurate method for measuring SOC.

Debates—Perspectives on socio‐hydrology: Socio‐hydrologic modeling: Tradeoffs, hypothesis testing, and validation

AbstractSocio‐hydrology focuses on studying the dynamics and co‐evolution of coupled human and water systems. Recently, several new socio‐hydrologic models have been published that explore these dynamics, and these models offer unique opportunities to better understand these coupled systems and to understand how water problems evolve similarly in different regions. These models also offer challenges, as decisions need to be made by the modeler on trade‐offs between generality, precision, and realism. In addition, traditional hydrologic model validation techniques, such as evaluating simulated streamflow, are insufficient, and new techniques must be developed. As socio‐hydrology progresses, these models offer a robust, invaluable tool to test hypotheses about the relationships between aspects of coupled human‐water systems. They will allow us to explore multiple working hypotheses to greatly expand insights and understanding of coupled socio‐hydrologic systems.

Modeling of the Interactions Between Catalytic Surfaces and Gas-Phase

The catalytic surface interacts with the gas-phase by a variety of chemical and physical processes. Hence, optimization of design and operation conditions of catalytic reactors do not only require the understanding of the catalytic reaction sequence but also its coupling with mass and heat transport and potential homogeneous reactions. The chemical, thermal, and mass-transport interactions between the catalytic surface and the gas-phase are discussed in terms of the individual and combined interactions. The state-of-the-art modelling of reactive flows and its coupling with the catalytic surface is summarized. The interactions are illustrated by a number of examples such as reforming of hydrocarbons, catalytic combustion, exhaust-gas after-treatment, each focusing on a special aspect of catalyst–gas interactions. The potentials and limitations of the numerical simulations will be discussed including experimental techniques for model validation.

PRM79 - Advishe: a New Tool to Report Validation of Health-Economic Decision Models

Background: Modelers and reimbursement decision makers could both profit from a more systematic reporting of the efforts to validate health-economic (HE) models. Objectives: Development of a tool to systematically report validation efforts of HE decision models and their outcomes. Methods: A gross list of model validation techniques was collected using a literature review, including sources outside the HE field. A panel then selected the most important items. Based on the Delphi method, the panel members could score items in three e-mail rounds. Participants were HE modelling experts, covering various nationalities and work environments. They could comment on relevance, feasibility and formulation of the items and received feedback on comments from others. This resulted in a draft tool of selected items, which was tested and improved in two further rounds. In addition, the Dutch National Health Care Institute commented on usefulness for decision makers, while a separate group of 50 HE experts could comment during a workshop at ISPOR Montreal 2014. Results: 35 Validation techniques were identified and grouped into four categories: conceptual model validation, computerized model validation, data validation and operational validation. Around 30 HE experts commented in each of the first three Delphi rounds, resulting in a 15 item draft tool. The Dutch health care advisory institute suggested to add one more item. Participants from the ISPOR workshop delivered 19 filled-in questionnaires. A fourth round resulted in 17 responses. This led to a refined version containing 16 items, which is currently sent out for a final, fifth round. Conclusions: When filled out by the modellers, AdVISHE (Assessment of the ValIdation Status of Health- Economic decision models) supports model users in assessing the validation status of a model It will be useful as part of reimbursement dossiers, by providing systematic and transparent insight into the validation efforts performed and their results.

Model Validation for Identification of Damage Dynamics

Robust control techniques have allowed engineers to create more descriptive models by including uncertainty in the form of additive noise and plant perturbations. As a result, the complete model set is robust to any discrepancies between the mathematical model and actual system. Experimental unfalsification of the model set leads to the guarantee that the model and uncertainties are able to recreate all experimental data points. In this work, such a robust control relevant model validation technique is applied to structural health monitoring in order to (1) detect the presence of damage and (2) identify the damage dynamics when used in conjunction with model-based identification. Additionally, the robust control relevant model validation technique allows for a novel quality measure of the identified damage dynamics. Feasibility of the method is demonstrated experimentally on a rotordynamic crack detection test rig with the detection and identification of a change in structure. Further insight is gained from application of the method to seeded damage on a rotor levitated on active magnetic bearings (AMBs) in the form of a local reduction in stiffness.

Receiver operating characteristics (ROC) as validation tool for prospectivity models — A magmatic Ni–Cu case study from the Central Lapland Greenstone Belt, Northern Finland

A prospectivity model for magmatic Ni–Cu deposits was created by integrating spatially referenced geophysical and geochemical datasets based on a simple and practical exploration model. The study area is the Central Lapland Greenstone Belt, Northern Fennoscandian Shield, Finland. Magmatic nickel deposits are related to rock types that are typically characterized by local magnetic and gravity anomalies. These deposit types can also be a source of nickel, copper and cobalt anomalies in the overlying glacial till cover. This straightforward exploration criterion was translated into a fuzzy logic prospectivity model. The model validation is an essential step in justifying the validity of the prospectivity model. This was accomplished by using receiver operating characteristics (ROC) technique. We used the known Ni–Cu occurrences and deposits as true positive cases and other deposit type locations or random points as true negative cases in the validation process. It appears that the ROC technique provides a robust model validation and optimization technique, providing that suitable validation data exists.

Wind turbine model validation: Fusion of simulation and measurement data

A new turbine model validation technique that is based on adaptation of look-up tables is described in this article. Simulation results from the VIDYN turbine simulation program and measurements from Big Glenn wind turbine, located outside Gothenburg, Sweden, are used as an input to this new model validation technique. The models of the flapwise bending moment and power coefficient are validated for Big Glenn turbine. Measurement data are acquired during normal turbine operation. Verification results show good agreement between model outputs and measured data. The method allows prediction in a wide range of turbine operating variables, using only few measured points.

Wind Turbine Model Validation: Fusion of Simulation and Measurement Data

A new turbine model validation technique that is based on adaptation of look-up tables is described this paper. Simulation results from the VIDYN turbine simulation program and measurements from Big Glenn wind turbine, located outside Gothenburg, Sweden are used as an input to this new model validation technique. The models of the flapwise bending moment and power coefficient are validated for Big Glenn turbine. Measurement data are acquired during normal turbine operation. Verification results show good agreement between model outputs and measured data. The method allows prediction in a wide range of turbine operating variables, using only few measured points.

A contribution to the development of a full flight envelope quasi-nonlinear helicopter simulation

Helicopter modeling is a complex task as helicopters consist of many subsystems (nacelle, rotor, engine, $$\ldots)$$ that are described by coupled differential equations. Controller design requires a model that is accurate over a certain frequency range. Usually, system identification is used to derive the models from flight test data and the model complexity depends on the frequency range of interest. As helicopters have eigenvalues that vary depending on airspeed, system identification is performed at different operating points. Each identified linear model is then valid only in a small region around the corresponding operating point. To arrive at a model that covers the whole operational envelope of the helicopter, the individual models are stitched together. This paper describes the derivation of a quasi-nonlinear model accounting for known nonlinear terms, such as gravity, inertia or trim curves. In addition, the coefficients of the identified linear matrices are interpolated depending on airspeed. Finally, the quasi-nonlinear as well as the linear model are compared against flight test data using model validation techniques. As the quasi-nonlinear model of the Flying Helicopter Simulator is validated, it can be used for a sophisticated controller design.

Gene regulatory network discovery using pairwise Granger causality

Discovery of gene regulatory network from gene expression data can yield a useful insight to drug development. Among the methods applied to time-series data, Granger causality (GC) has emerged as a powerful tool with several merits. Since gene expression data usually have a much larger number of genes than time points therefore a full model cannot be applied in a straightforward manner, GC is often applied to genes pair wisely. In this study, the authors first investigate with synthetic data how spurious causalities (false discoveries) may arise because of the use of pairwise rather than full-model GC detection. Furthermore, spurious causalities may also arise if the order of the vector autoregressive model is not high enough. As a remedy, the authors demonstrate that model validation techniques can effectively reduce the number of false discoveries. Then, they apply pairwise GC with model validation to the real human HeLa cell-cycle dataset. They find that Akaike information criterion is generally most suitable for determining model order, but precaution should be taken for extremely short time series. With the authors proposed implementation, degree distributions and network hubs are obtained and compared with existing results, giving a new observation that the hubs tend to act as sources rather than receivers of interactions.

Teaching ‘Instant Experience’ with Graphical Model Validation Techniques

AbstractGraphical model validation techniques for linear normal models are often used to check the assumptions underlying a statistical model. We describe an approach to provide ‘instant experience’ in looking at a graphical model validation plot, so it becomes easier to validate if any of the underlying assumptions are violated.

Evaluation of Model Validation Techniques in Land Cover Dynamics

This paper applies different methods of map comparison to quantify the characteristics of three different land change models. The land change models used for simulation are termed as “Stochastic Markov (St_Markov)”, “Cellular Automata Markov (CA_Markov)” and “Multi Layer Perceptron Markov (MLP_Markov)” models. Various model validation techniques such as per category method, kappa statistics, components of agreement and disagreement, three map comparison and fuzzy methods have then been applied. A comparative analysis of the validation techniques has also been discussed. In all cases, it is found that “MLP_Markov” gives the best results among the three modeling techniques. Fuzzy set theory is the method that seems best able to distinguish areas of minor spatial errors from major spatial errors. Based on the outcome of this paper, it is recommended that scientists should try to use the Kappa, three map comparison and fuzzy methods for model validation. This paper facilitates communication among land change modelers, because it illustrates the range of results for a variety of model validation techniques and articulates priorities for future research.

Statistical emulators of a plant growth simulation model

CR Climate Research Contact the journal Facebook Twitter RSS Mailing List Subscribe to our mailing list via Mailchimp HomeLatest VolumeAbout the JournalEditorsSpecials CR 55:253-265 (2013) - DOI: https://doi.org/10.3354/cr01138 Statistical emulators of a plant growth simulation model Padmaja Ramankutty1,2,3,*, Megan Ryan1,2, Roger Lawes4, Jane Speijers3, Michael Renton1,2,4 1School of Plant Biology and Institute of Agriculture, and 2Future Farm Industries CRC, The University of Western Australia, 35 Stirling Hwy, Crawley, Western Australia 6009, Australia 3Department of Agriculture and Food, 3 Baron-Hay Court, South Perth, Western Australia 6151, Australia 4CSIRO Ecosystem Sciences, Private Bag 5, Wembley, Western Australia 6913, Australia *Email: ramanp01@student.uwa.edu.au ABSTRACT: A new methodology was developed to create statistical emulators from an existing complex plant growth simulation model, in order to provide simple models for predicting perennial pasture production based on meteorological data. The goal was to find emulators that could fit simulation data and provide reliable predictions of new simulation output. Four types of statistical emulators were compared—linear models, non-linear models, linear splines and cubic splines—which varied in complexity due to the use of different functional forms to fit main effects and different methods to determine model parameters. Two methods of model validation were used to ensure that the emulator selected provided good fits to the initial simulation output and gave adequate predictions of future simulation output. We illustrated the various techniques of model development and validation using the Agricultural Production Systems sIMulator (APSIM) plant growth model. The effects of rainfall amount, rainfall frequency, temperature and radiation on APSIM-generated biomass production of the perennial pasture species lucerne Medicago sativa were used as a case study. We conclude that, in this case, the best statistical emulator is the cubic spline. The novel approach illustrated here provides a rigorous, flexible and powerful means of developing and testing emulators for any complex simulation model. KEY WORDS: Statistical emulator · Linear mixed effects model · Model validation · Cubic splines · APSIM · Medicago sativa · Biomass production Full text in pdf format Supplementary material PreviousCite this article as: Ramankutty P, Ryan M, Lawes R, Speijers J, Renton M (2013) Statistical emulators of a plant growth simulation model. Clim Res 55:253-265. https://doi.org/10.3354/cr01138 Export citation RSS - Facebook - Tweet - linkedIn Cited by Published in CR Vol. 55, No. 3. Online publication date: February 06, 2013 Print ISSN: 0936-577X; Online ISSN: 1616-1572 Copyright © 2013 Inter-Research.

Applicability of Monte Carlo cross validation technique for model development and validation using generalised least squares regression

Summary In regional hydrologic regression analysis, model selection and validation are regarded as important steps. Here, the model selection is usually based on some measurements of goodness-of-fit between the model prediction and observed data. In Regional Flood Frequency Analysis (RFFA), leave-one-out (LOO) validation or a fixed percentage leave out validation (e.g., 10%) is commonly adopted to assess the predictive ability of regression-based prediction equations. This paper develops a Monte Carlo Cross Validation (MCCV) technique (which has widely been adopted in Chemometrics and Econometrics) in RFFA using Generalised Least Squares Regression (GLSR) and compares it with the most commonly adopted LOO validation approach. The study uses simulated and regional flood data from the state of New South Wales in Australia. It is found that when developing hydrologic regression models, application of the MCCV is likely to result in a more parsimonious model than the LOO. It has also been found that the MCCV can provide a more realistic estimate of a model’s predictive ability when compared with the LOO.

Validation of System Dynamics Models – a Case Study

The purpose of this article is the analysis of the system dynamics model validation illustrated by the example of a model of the manufacturing resource allocation. In the first part of the article the authors present an overview of the definitions of validation and verification that can be found in the reference literature. Also, they emphasize the role which validation and verification play in the modeling process. Furthermore, they discuss the techniques of system dynamics model validation with particular focus on tests of the model structure, behavior and policy implications. The second part of the article contains an example of the validation process of a system dynamics model simulating manufacturing resource allocation in an electronic company. The purpose of the model is to assess the long-term effect of assigning workers to individual tasks on such production process parameters as efficiency or effective work time. The authors focus their particular attention on that part of the model which deals with a storehouse, one of the company production units. They conduct tests of its structure and behavior. When validating the structure the authors make use of the information obtained in a series of interviews with the company staff. They also refer to the generally accepted knowledge found in the reference literature. The results generated by the model in the course of the behavior tests are compared with the real data. The authors evaluate both the logic of the system behavior and the level of accuracy of the output data in reference to the real system.

A comparison of internal model validation methods for multifactor dimensionality reduction in the case of genetic heterogeneity

BackgroundDetermining the genes responsible for certain human traits can be challenging when the underlying genetic model takes a complicated form such as heterogeneity (in which different genetic models can result in the same trait) or epistasis (in which genes interact with other genes and the environment). Multifactor Dimensionality Reduction (MDR) is a widely used method that effectively detects epistasis; however, it does not perform well in the presence of heterogeneity partly due to its reliance on cross-validation for internal model validation. Cross-validation allows for only one “best” model and is therefore inadequate when more than one model could cause the same trait. We hypothesize that another internal model validation method known as a three-way split will be better at detecting heterogeneity models.ResultsIn this study, we test this hypothesis by performing a simulation study to compare the performance of MDR to detect models of heterogeneity with the two different internal model validation techniques. We simulated a range of disease models with both main effects and gene-gene interactions with a range of effect sizes. We assessed the performance of each method using a range of definitions of power.ConclusionsOverall, the power of MDR to detect heterogeneity models was relatively poor, especially under more conservative (strict) definitions of power. While the overall power was low, our results show that the cross-validation approach greatly outperformed the three-way split approach in detecting heterogeneity. This would motivate using cross-validation with MDR in studies where heterogeneity might be present. These results also emphasize the challenge of detecting heterogeneity models and the need for further methods development.

A PETRI NET BASED AGENT BEHAVIORAL TESTING

In Multi-Agent System (MAS), developers concentrate on creating design models and evolving them, from higher level models to lower level models, in several steps. Considerable part of MAS implementations is automatically produced from the design models. If a design model contains faults, they are passed to the generated implementations. Practical model validation techniques are required to discover and delete faults in abstract design models. We introduce a formal approach for agent design testing. It specifies a testing process that complements Multi-agent Systems Engineering (MaSE) methodology and strengthens the mutual relationship between UML and MAS. Besides, it defines a structured and comprehensive testing process for engineering software agents at the design level by providing a systematic way of converting the MAS design models to UML design diagram. Petri Net (PN) diagram is generated from the UML models to simulate the behavior of an agent. Because Petri Nets (PNs) are formal models, their analysis techniques can be applied to automatic agent behavioral testing.

Modeling and Predicting Cumulative Population of Bangladesh

The purpose of the present study is to build up some mathematical models for male and female forward cu- mulative population of Bangladesh. For this, the secondary data for population of Bangladesh have been taken from census of 2001. To check up the validity of the models, the model validation technique, cross-validity prediction power (CVPP), is employed in this paper. It is seen that male and female forward cumulative population of Bangladesh follow four parameters polynomial model. It is found that the parameters of these fitted models are highly significant with large proportion of variance explained. Moreover, these models are stable more than 99%. The stability of R 2 of these models is more than 99%. Hence, the fittings of these models are well.

Semiparametric Hidden Markov Models

Hidden Markov models (HMMs) are widely used for dependent data modeling. Classically, the state-dependent distributions, that is, the distribution of the observation given the hidden state, belong to some parametric family. This article proposes a semiparametric HMM for continuous observations with at least one state-dependent distribution modeled with modern nonparametric techniques, for example, nonparametric maximum likelihood estimation under shape constraints like log-concavity. The resulting model is much more flexible and avoids a model misspecification bias. Concentrating on the special case of two states, the article discusses identifiability of the model. Further, an estimation procedure for semiparametric HMMs is proposed based on the expectation–maximization algorithm, which gives also rise to model validation techniques, as demonstrated in a simulation study and an empirical illustration.