Verification Of Simulation Models Research Articles

Guide to credible computer simulations of fluid flows

The significance of computer simulations depends solely on their credibility. A user of computer products, simulations and software, expects that these products are credible for the intended use. Because no standards exist for fluid-flow simulations by which to establish this credibility, a guide is presented here. The credibility is established by conducting verification and validation of simulation models and certification of simulations and of simulation software. Verification assesses whether the problem is solved correctly and estimates the level of computational accuracy of simulations; validation determines whether the right problem is solved and assesses the level of the validity of the simulation model by estimating the degree to which simulations accurately represent reality. These processes are achieved by identifying the intended uses of the simulations and the sources of uncertainties in them and by conducting sensitivity-uncertainty analyses. Certification determines 1) whether a software in terms of its logic, conceptual and computational models, procedures, rules, and documentation and 2) whether the simulations derived from the software are in compliance with specified requirements.

Verification and validation of simulation models

This paper surveys verification and validation of models, especially simulation models in operations research. For verification it discusses 1) general good programming practice (such as modular programming), 2) checking intermediate simulation outputs through tracing and statistical testing per module, 3) statistical testing of final simulation outputs against analytical results, and 4) animation. For validation it discusses 1) obtaining real-worl data, 2) comparing simulated and real data through simple tests such as graphical, Schruben-Turing, and t tests, 3) testing whether simulated and real responses are positively correlated and moreover have the same mean, using two new statistical procedures based on regression analysis, 4) sensitivity analysis based on design of experiments and regression analysis, and risk or uncertainty analysis based on Monte Carlo sampling, and 5) white versus black box simulation models. Both verification and validation require good documentation, and are crucial parts of assessment, credibility, and accreditation. A bibliography with 61 references is included.

The role of long-term sites in agroecological research: A case study

Gradual, episodic or subtle changes in agroecosystems can be definitively measured only in long-term studies. The objective of this paper is to illustrate the benefits and limitations of long-term agroecological sites using a crop rotation study (Rotation ABC) established at Lethbridge, Alberta in 1910. Long-term agroecological sites, envisioned as part of a continuum of research levels, seek to determine the persistence of cropping systems with respect to their effects on soil resources and the larger environment. Examples of research findings arising from such sites include the documentation of long-term changes in agricultural productivity, measurement of organic matter changes over time, calculation of long-term nutrient budgets, and verification of simulation models. For example, soil organic matter concentrations in Rotation ABC were found to decline abruptly within the first few decades of cultivation and then approach a new steady state. Despite the continual removal of N in harvested grain, soil N concentrations also appear to have stabilized, suggesting an accession of N approximately equivalent to the amount harvested. Common limitations of long-term sites like that at Lethbridge include: rudimentary experimental design, evolving management practices, incidental changes at the sites, and variation in sampling protocols. The successful use of long-term agroecological sites depends on well-defined objectives, simplicity of design, adequate scale, consistent data and sample collection, and collaboration among sites. The primary justification for the establishment and maintenance of long-term agroecological sites may be their future role in answering questions not yet envisioned. Key words: Soil organic matter, long-term studies, crop rotations, sustainability, fertilizer

Thermal energy storage in an unconfined aquifer: 1. Field Injection Experiment

A thermal injection and storage experiment was conducted to investigate the feasibility of storing thermal energy in shallow unconfined aquifers near the water table. Heated water was injected into a shallow aquifer and plume temperatures were monitored over a 141‐day period by means of a dense array of bundle‐type piezometers. The highly detailed data, which provide the three‐dimensional temperature distribution within the aquifer, give good insight into the physical processes of aquifer thermal energy storage, and provide an excellent basis for the verification of simulation models. The experimental data also allow the physical processes of heat advection, dispersion, retardation, buoyancy and boundary heat loss to be quantified. In a companion paper, a three‐dimensional density‐dependent groundwater flow and thermal transport model is developed and validated using the results of the thermal injection experiment.

Analytic performance estimation for complex maintenance systems

Earlier models are extended and combined to provide an analytic means of estimating the average number of inoperative machines in a two-echelon (bases and a depot), two-indenture (machines have components which may have modules) maintenance system with spares and repair. Separate expressions are developed for the no cannibalization and complete, instantaneous cannibaliza tion cases. The expressions are suitable for verification of aircraft maintenance system simulation models as well as for estima tion of system performance and the maximum benefit of cannibalization.

An expert system for verification and tuning of simulation models: Renato de Mori and Robert PragerSchool of Computer Science, McGill University, Montreal, Quebec, Canada

An expert system for verification and tuning of simulation models: Renato de Mori and Robert PragerSchool of Computer Science, McGill University, Montreal, Quebec, Canada

Numerical Simulation and Experimental Verification of the German Roller Rig for Rail Vehicles*

SUMMARY A numerical simulation model of the roller test stand located at Munich and loaded by a bogie is discussed including its technical structure, the governing physical equations of motion and the structure of the simulation program. Both, the set up of the mathematical and numerical models time and the computation time of simulation runs have been considerably reduced (by a factor of 20) using formula manipulation programs. Simulation results concerning the starting behaviour of a bogie, stationary limit cycle behaviour of bogies with ideal and wear profile and influence of gauge changes and spring/damper modifications on limit cycle behaviour of a bogie are presented, some of which are compared with experimental results gained from test facility measurements. The simulation results are in good agreement with the experimental results and provide an experimental verification of the roller rig simulation model presented.

The Verification of Dynamic Simulation Models

This paper is concerned with the use of ‘loop analysis’ in the evaluation and validation of dynamic simulation models. The part that loop analysis plays in the overall validation process is described, and it is shown, by reference to the analysis of a complex model, how the model's behaviour may be investigated. The analysis proceeds by identifying some of the elements of system behaviour such as damping, phase-shifts and gain/delay factors, and, in the process, highlights those feedback loops that are pseudo-positive and/or unconformable. In addition to the validation aspect of the use of loop analysis, there is a valuable contribution towards the knowledge and insight of the model-builder, enabling him better to identify areas of the system which may benefit from re-design.

The Verification of Dynamic Simulation Models

The Verification of Dynamic Simulation Models

An Experimental Study of Upper Hot Layer Stratification in Full-Scale Multiroom Fire Scenarios

This paper describes an experimental study of the dynamics of smoke filling in realistic, full-scale, multiroom fire scenarios. A major objective of the study was to generate an experimental data base for use in the verification of mathematical fire simulation models. The test space involved 2 or 3 rooms, connected by open doorways. During the course of the study the areas were partitioned to yield four different configurations. One of the rooms was a burn room containing a methane burner which produced either a constant energy release rate of 25, 100, or 225 kW or a time-varying energy release rate which increased linearly with time from zero at ignition to 300 kW in 10 min. An artificial smoke source near the ceiling of the burn room provided a means for visualizing the descent of the hot layer and the dynamics of the smoke filling process in the various spaces. The development of the hot stratified layers in the various spaces was monitored by vertical arrays of thermocouples and photometers. A layer interface was identified and its position as a function of time was determined. An analysis and discussion of these results are presented.

Development and verification of simulation models for describing pesticide dynamics in soils

Mathematical models currently available for describing pesticide dynamics in soils were reviewed. Processes included in this review were: retention (adsorption-desorption), transformations (chemical and/or biological degradation), and convective-dispersive transfort with soil-water. A large number of simulation models, with varying degrees of complexity and scope, are presently available. However, verification and extensive use of these models has generally been difficult owing to inadequate methods of measuring and/or estimating the necessary model input parameters. The ability to formulate and numerically solve complex simulation models apparently exceeds our ability to verify these models using the limited experimental data base presently available. Given the uncertainties of model parameters and the field-scale soil heterogeneity, deviations of a factor of two or more between measured and simulated results may be common. Therefore, for general use, simple models with a minimum number of inputs may provide a sufficiently accurate description of pesticide dynamics in agricultural ecosystems. Only limited amounts of field data are presently available to verify even the simple process-orientated models.

Statistical evaluation and verification of digital simulation models

In most instances, designers of digital simulation models ignore the statistical design and the statistical implications associated with these models. Most ignore the problem because of the added difficulties related to autocorrelated data. Fishman and Kiviat of the RAND Corporation [33] suggested the use of spectral analysis as a technique for evaluating the significance of statistics emanating from a simulation model, comparing model output with “real-world” data for model verification and comparing model outputs for two or more policy change evaluations. The authors have developed this concept into a synthesized model and verified the model through a spectral analysis on data derived from the Logistics Composite Model (LCOM). LCOM was developed jointly by the RAND Corporation and the Air Force Logistics Command for the purpose of simulating aircraft flight and base support processes in response to mission requirements. Some forty statistics were evaluated through spectral analysis for confidence limits. Further, certain policy changes regarding inventory re-order were implemented and a spectral analysis was performed to judge whether or not the separate results were statistically different.

A Note on Verification of Computer Simulation Models

This paper establishes an argument that questions the validity of one “test” of goodness-of-fit for the simulated time path of a single endogenous variable in a simultaneous, perhaps dynamic, econometric model. The test was suggested by Cyert and Cohen, and consists of two parts, within the context of a regression of the actual series on the generated series: a test that the intercept of this regression differs significantly from zero and a test that the slope of this regression differs significantly from one. Presumably, the intuition underlying the test is that if the simulation model is a good one this regression should be a 45° line through the origin. The paper's primary purpose is to demonstrate that this intuition is wrong in general for the case of “stochastic simulation.”

Some Practical Aspects of the Verification and Validation of Simulation Models

Some Practical Aspects of the Verification and Validation of Simulation Models

Some Practical Aspects of the Verification and Validation of Simulation Models

This paper reviews the philosophical questions of model validity and describes the need for practical tests of the adequacy and representativeness of computerized simulation models. The distinction between the concepts of the verification and the validation of such general models is made by means of a delineation of test procedures, primarily those of a statistical nature, which are applicable to the determination of a simulation model's adequacy.

Critique of: “Verification of Computer Simulation Models”

Critique of Naylor, T. H., J. M. Finger. 1967. Verification of computer simulation models. Management Sci. 14 (2, October) B-92–B-101.

Critique of: “Verification of Computer Simulation Models”

Critique of Naylor, T. H., J. M. Finger. 1967. Verification of computer simulation models. Management Sci. 14 (2, October) B-92–B-101.

Verification of Computer Simulation Models

The problem of validating computer simulation models of industrial systems has received only limited attention in the management science literature. The purpose of this paper is to consider the problem of validating computer models in the light of contemporary thought in the fields of philosophy of science, economic theory, and statistics. In order to achieve this goal we have attempted to gather together and present some of the ideas of scientific philosophers, economists, statisticians, and practitioners in the field of simulation which are relevant to the problem of verifying simulation models. We have paid particular attention to the writings of economists who have been concerned with testing the validity of economic models. Among the questions which we shall consider are included: What does it mean to verify a computer model of an industrial system? Are there any differences between the verification of computer models and the verification of other types of models? If so, what are some of these differences? Also considered are a number of measures and techniques for testing the “goodness of fit” of time series generated by computer models to observed historical series.