Model Validation Method Research Articles

The General Principles of the Transportation Simulation Model Development and Validation

Transportation simulation model development allows simulating traveller’s decisions, evaluating various transportation management strategies and complex solutions. The aim of the paper is to set the general principles of the transportation simulation model development and validation. The paper contains the overview of the transportation simulation models types with the examples from the conducted projects for the Riga city. The basic steps of the simulation model development procedure: initial data preparation and analysis, transportation model development and simulation, scenarios planning and evaluation, and simulation models outcomes evaluation are considered. Simulation model verification, validation and calibration definitions are given. The basic checks for the transportation macroscopic and microscopic simulation model validation are listed. A summary of the transportation simulation model validation and calibration methods and parameters is given.

An adaptive sequential experiment design method for model validation

Efficient experiment design is of great significance for the validation of simulation model with high nonlinearity and large input space. Excessive validation experiment raises the cost while insufficient test increases the risks of accepting an invalid model. In this paper, an adaptive sequential experiment design method combining global exploration criterion and local exploitation criterion is proposed. The exploration criterion utilizes discrepancy metric to improve the space-filling property of the design points while the exploitation criterion employs the leave one out error to discover informative points. To avoid the clustering of samples in the local region, an adaptive weight updating approach is provided to maintain the balance between exploration and exploitation. Besides, the credibility distribution function characterizing the relationship between the input and result credibility is introduced to support the model validation experiment design. Finally, six benchmark problems and an engineering case are applied to examine the performance of the proposed method. The experiments indicate that the proposed method achieves satisfactory performance for function approximation in accuracy and convergence.

Uncertainty quantification and statistical model validation for an offshore jacket structure panel given limited test data and simulation model

Due to inherent variability (i.e., aleatory uncertainty) in material properties, loading conditions, manufacturing processes, etc., simulation output responses should follow certain distributions, which are the outcome of input distribution models and simulation models. Thus, uncertainty quantification (UQ) using simulation-based methods is accurate only if we have (1) accurate input distribution models and (2) an accurate simulation model. However, in practical engineering applications, only limited numbers of input test data are available for modeling input distributions. Moreover, the simulation model could be biased due to assumptions and idealizations in the modeling process. Thus, the simulation model needs to be validated to correctly predict the output response of the physical system. The statistical validation of the simulation model would require large numbers of physical test data, which is extremely expensive. This study presents a computational method to obtain a target output distribution, which is a good approximation of the true output distribution given limited test data and a biased model. Using Bayesian analysis, possible candidates of output distribution are obtained, and a target output distribution is selected at the posterior median. The target output distribution is used to measure the bias of the simulation models and the surrogate models for UQ and statistical model validation. Furthermore, the cumulative distribution function of the simulation model prediction error is presented to provide (1) the median of model prediction error and (2) model confidence at a user-specified error level. To demonstrate the proposed statistical model validation method, a simulation model of an offshore jacket structure panel is generated using ANSYS. This example has eight uncertain input parameters (six of them are geometrical parameters and two of them are material properties). For input distributions of geometrical parameters, the standard deviation of manufacturing tolerances is used, and three material tests are carried out to obtain input distributions of two material properties. The output response of interest is a reaction force. The simulation model is validated using three sets of material test data and experimental test data to determine model confidence on output prediction and the validity of the simulation model. The result of statistical model validation shows how much bias the simulation model has and how much confidence engineers can have in the model prediction at a user-specified error level.

The Knowing Model: Facilitating Behaviour Change in Organisations

It is incumbent upon leadership to encourage member behaviour that is beneficial to both the organisation and the members themselves. This paper outlines a context-based model — utilising the existing theories and models in Knowledge Management, Library and Information Science, and Public Health — to increase leadership’s effectiveness in this area. The Knowing Model approaches behaviour change as an issue of information content, dissemination, and use — all within a complex environment. A behaviour — one that an organisation has unsuccessfully attempted to change in the past — is identified by leadership. Leaders then analyse the extent to which members were aware of these previous efforts, integrated information about them, and acted because of them. This data is used to develop an information campaign aimed at changing the target behaviour. Because organisations likely have several information dissemination channels available to them, this model is offered as a low-cost and low-resource first step in behaviour change efforts. It will not replace more extensive efforts noted in previous models, but because of its theoretical grounding it is expected to promote change in certain areas. Approaches requiring more extensive resources can then be applied to other areas. This paper outlines the Knowing Model, strategies for information dissemination, and methods for model implementation and validation.

Validation Method for Simulation Model of Internet of Things-Aided Power System

Validation of power system simulation models is essentially a similarity analysis problem based on multivariate time series. With the development of the internet of things (IoT) technology in the power system, the interoperability and integration of devices in the practical project are improved, and the cross interaction in the simulation process becomes more complex correspondingly. It is critical to explore the inherent correlation from the high dimensional data to evaluate the credibility and to locate the error of the simulation model. Thus, a model validation method based on factor analysis and the Prony method is proposed in this paper. Firstly, the multivariate time series of the simulation model and the practical/acknowledged system are replaced by a low number of common factors with physical meanings by factor analysis. Secondly, the modified adaptive Prony method is applied to extract the features of each common factor to ensure the best fitting of the non-stationary signal. Then the complete similarity evaluation model of the simulation system is established based on energy proportion, information entropy, and variance of the contribution rate. Finally, the error location is identified in the evaluation process based on the physical meaning of extracted features. The feasibility and effectiveness of the proposed method are verified by an application in the simulation model of a power electronics system developed in PSASP.

Model Validation Methods for Multiple Correlated Responses via Covariance-Overlap Based Distance

Abstract Model validation methods have been widely used in engineering design to provide a quantified assessment of the agreement between simulation predictions and experimental observations. For the validation of simulation models with multiple correlated outputs, not only the uncertainty of the responses but also the correlation between them needs to be considered. Most of the existing validation methods for multiple correlated responses focus on the area metric, which only compares the overall area difference between the two cumulative probability distribution curves. The differences in the distributions of the data sets are not fully utilized. In this paper, two covariance-overlap based model validation (COMV) methods are proposed for the validation of multiple correlated responses. The COMV method is used for a single validation site, while the covariance-overlap pooling based model validation (COPMV) method can pool the evidence from different validation sites into a scalar measure to give a global evaluation about the candidate model. The effectiveness and merits of the proposed methods are demonstrated by comparing with three different existing validation methods on three numerical examples and a practical engineering problem of a turbine blade validation example. The influence of sample size and the number of partitions in the proposed methods are also discussed. Results show that the proposed method shows better performance on the uncertainty estimation of different computational models, which is useful for practical engineering design problems with multiple correlated responses.

Residential Location Choices of Couples Considering both Partners’ Residential Biographies and Family Ties

We investigate couples’ residential decisions by considering variables capturing elements of both partners’ residential biographies and family ties. We focus on the family formation stage because decisions made in this stage are rather long-term. We are particularly interested in the hometown as individuals have spent a great amount of time and an important life phase there. Our research questions are: (1) To what extent do people live in their hometown at family formation? (2) Which factors influence this choice? We consider previous residential experiences, social ties, socio-demographic and spatial variables in binary logit regressions. The data was collected from a sample of students at TU Dortmund University, their parents, and grandparents. In this paper, the parent couples’ residential location choices are analysed considering family ties to the grandparents. We apply a rigorous split half method for internal model validation. Recent research on residential decisions mostly considers either the partners or the life-course or social ties. Due to our unique dataset, we include all these factors. This allows us to draw conclusions on gendered residential choice. We find the length of residence to be positively and number of moves negatively associated with living at the hometown, two factors that have not been disentangled in previous research. We could not confirm the often proven male dominance. On the contrary, we found that the female’s family ties were more relevant. The hometown has not received the attention it deserves to date and has the potential to enrich demographic research and related fields. The rigorous split half method for internal validation has rarely been undertaken previously. The results point to the importance of model validation and thus present an innovative approach.

NTCP model validation method for DAHANCA patient selection of protons versus photons in head and neck cancer radiotherapy

Introduction: Prediction models using logistic regression may perform poorly in external patient cohorts. However, there is a need to standardize and validate models for clinical use. The purpose of this project was to describe a method for validation of external NTCP models used for patient selection in the randomized trial of protons versus photons in head and neck cancer radiotherapy, DAHANCA 35.Material and methods: Organs at risk of 588 patients treated primarily with IMRT in the randomized controlled DAHANCA19 trial were retrospectively contoured according to recent international recommendations. Dose metrics were extracted using MatLab and all clinical parameters were retrieved from the DAHANCA database. The model proposed by Christianen et al. to predict physician-rated dysphagia was validated through the closed testing, where change of the model intercept, slope and individual beta’s were tested for significant prediction improvements.Results: Six months prevalence of dysphagia in the validation cohort was 33%. The closed testing procedure for physician-rated dysphagia showed that the Christianen et al. model needed an intercept refitting for the best match for the Danish patients. The intercept update increased the risk of dysphagia for the validation cohort by 7.9 ± 2.5% point. For the raw model performance, the Brier score (mean squared residual) was 0.467, which improved significantly with a new intercept to 0.415.Conclusions: The previously published Dutch dysphagia model needed an intercept update to match the Danish patient cohort. The implementation of a closed testing procedure on the current validation cohort allows quick and efficient validation of external NTCP models for patient selection in the future.

The R2eD Arteriovenous Malformation Score: A Novel Predictive Tool for Arteriovenous Malformation Presentation With Hemorrhage

Abstract INTRODUCTION The management of unruptured brain arteriovenous malformations (AVMs) remains unclear. This study aims to develop a predictive tool that could guide hemorrhage risk stratification. METHODS A database of 789 AVM patients presenting to our institution between 1990 and 2017 was used. A hold-out method of model building and validation was employed whereby the data was randomly split in half into training and validation datasets. Factors significantly associated with hemorrhage presentation at the univariable level in the training dataset were used to construct a model based on multivariable logistic regression. Model performance was assessed using receiver operating curves (ROC) on the training, validation, and complete datasets. The model predictors and the complete dataset were then used to derive a formula for risk prediction and a practical scoring system according to the model coefficients. RESULTS In 755 patients with complete data on presentation status, 272 (36%) presented with hemorrhage. After model building and validation, the final model contained the following risk factors: non-white race (odds ratio [OR] = 2.49, P < .01), deep location (OR = 1.68, P = .02), small AVM size (<3 cm, OR = 1.63, P < .01), exclusive deep venous drainage (OR = 1.73, P = .02), and “monoarterial” feeding (OR = 1.58, P = .02). Area under the curve from ROC analysis was 0.702, 0.698, and 0.685 for the training, validation, and complete datasets, respectively. Every risk factor was worth 1 point except race, which was worth 2 points. The factors are summarized by the acronym R2eD AVM (total score varies from 0 to 6). In the entire study population, the predicted probability of hemorrhagic presentation increased in a stepwise manner from 16% for patients with no risk factors (score of 0) to 78% for patients having all the risk factors (score of 6) with around a 10% increase in risk per added point. CONCLUSION This risk can be used as a predictive tool that supplements clinical judgement and aids in patient counselling.

Modeling the Risk of Team Sport Injuries: A Narrative Review of Different Statistical Approaches.

Injuries are a common occurrence in team sports and can have significant financial, physical and psychological consequences for athletes and their sporting organizations. As such, an abundance of research has attempted to identify factors associated with the risk of injury, which is important when developing injury prevention and risk mitigation strategies. There are a number of methods that can be used to identify injury risk factors. However, difficulty in understanding the nuances between different statistical approaches can lead to incorrect inferences and decisions being made from data. Accordingly, this narrative review aims to (1) outline commonly implemented methods for determining injury risk, (2) highlight the differences between association and prediction as it relates to injury and (3) describe advances in statistical modeling and the current evidence relating to predicting injuries in sport. Based on the points that are discussed throughout this narrative review, both researchers and practitioners alike need to carefully consider the different types of variables that are examined in relation to injury risk and how the analyses pertaining to these different variables are interpreted. There are a number of other important considerations when modeling the risk of injury, such as the method of data transformation, model validation and performance assessment. With these technical considerations in mind, researchers and practitioners should consider shifting their perspective of injury etiology from one of reductionism to one of complexity. Concurrently, research implementing reductionist approaches should be used to inform and implement complex approaches to identifying injury risk. However, the ability to capture large injury numbers is a current limitation of sports injury research and there has been a call to make data available to researchers, so that analyses and results can be replicated and verified. Collaborative efforts such as this will help prevent incorrect inferences being made from spurious data and will assist in developing interventions that are underpinned by sound scientific rationale. Such efforts will be a step in the right direction of improving the ability to identify injury risk, which in turn will help improve risk mitigation and ultimately the prevention of injuries.

Deep Neural Network-Based Gait Classification Using Wearable Inertial Sensor Data.

Human gait has been regarded as a useful behavioral biometric trait for personal identification and authentication. This study aimed to propose an effective approach for classifying gait, measured using wearable inertial sensors, based on neural networks. The 3-axis accelerometer and 3-axis gyroscope data were acquired at the posterior pelvis, both thighs, both shanks, and both feet while 29 semi-professional athletes, 19 participants with normal foot, and 21 patients with foot deformities walked on the 20-meter straight path. The classifier based on the gait parameters and fully connected neural network was developed by applying 4-fold cross-validation to 80% of the total samples. For the test set that consisted of the remaining 20% of the total samples, this classifier showed an accuracy of 93.02% in categorizing the athlete, normal foot, and deformed foot groups. Using the same model validation and evaluation method, up to 98.19% accuracy was achieved from the convolutional neural network-based classifier. This classifier was trained with the gait spectrograms obtained from the time-frequency domain analysis of the raw acceleration and angular velocity data. The classification based only on the pelvic spectrograms exhibited an accuracy of 94.25% even without requiring a time-consuming and resource-intensive process for feature engineering. The notable performance and practicality in gait classification achieved by this study suggest potential applicability of the proposed approaches in the field of biometrics.

Validation method for simulation models with cross iteration

Cross iteration often exists in the computational process of the simulation models, especially for control models. There is a credibility defect tracing problem in the validation of models with cross iteration. In order to resolve this problem, after the problem formulation, a validation theorem on the cross iteration is proposed, and the proof of the theorem is given under the cross iteration circumstance. Meanwhile, applying the proposed theorem, the credibility calculation algorithm is provided, and the solvent of the defect tracing is explained. Further, based on the validation theorem on the cross iteration, a validation method for simulation models with the cross iteration is proposed, which is illustrated by a flowchart step by step. Finally, a validation example of a six-degree of freedom (DOF) flight vehicle model is provided, and the validation process is performed by using the validation method. The result analysis shows that the method is effective to obtain the credibility of the model and accomplish the defect tracing of the validation.

Principles and methods of model validation for model risk reduction

Models, being incomplete and simplified representations of complex, real-world systems, contain simplifications and abstractions. However, models with errors or incorrect assumptions can suggest misleading results from the perspective of the decision-maker. Model risk arises from errors within models, or the incorrect use of models, and if undetected, can have significant impacts on businesses and organizations whose decisions and business processes depend on the model outputs. Thus, there is a need for model validation to reduce model risk. Validation is a process of comparing the correspondence between model outputs and system behavior. Systems engineering, given its primary focus on the theory and methodology of systems modeling, is a valuable source for principles and best practices in model validation, even for applications outside of systems engineering. In this paper, we discuss how model validation principles and methods developed in systems engineering research are applicable across a variety of modeling methodologies and domain areas for the reduction of model risk. Specific open challenges are discussed, including dealing with the subjectivity in models, the validation of decision models, and determining what level of validation is sufficient.

High Resolution Urban Air Quality Modeling by Coupling CFD and Mesoscale Models: a Review

According to World Health Organization, 9 out of 10 people breathe polluted air and the ambient air pollution accounts for nearly 4.2 million early deaths worldwide. There is an urgent need for scientific management of urban air systems. Mathematical modeling of air quality helps the researchers and urban authorities in devising scientific management plans for mitigation of the associated impacts. We present an organized review of the broad aspects related to urban air quality modeling such as – urban microclimate, geospatial data, chemical transport models, computational fluid dynamics (CFD) models and integration of CFD and mesoscale models. The paper also discusses about the influence of urban land scape features on air quality, accuracy of emission inventory and model validation methods. The present review provides a vantage point to the researchers in the emerging field of high resolution urban air quality modeling for devising the location specific mitigation plans for the scientific management of the clean air.

Benchmarking CME Arrival Time and Impact: Progress on Metadata, Metrics, and Events

AbstractAccurate forecasting of the arrival time and subsequent geomagnetic impacts of coronal mass ejections (CMEs) at Earth is an important objective for space weather forecasting agencies. Recently, the CME Arrival and Impact working team has made significant progress toward defining community‐agreed metrics and validation methods to assess the current state of CME modeling capabilities. This will allow the community to quantify our current capabilities and track progress in models over time. First, it is crucial that the community focuses on the collection of the necessary metadata for transparency and reproducibility of results. Concerning CME arrival and impact we have identified six different metadata types: 3‐D CME measurement, model description, model input, CME (non)arrival observation, model output data, and metrics and validation methods. Second, the working team has also identified a validation time period, where all events within the following two periods will be considered: 1 January 2011 to 31 December 2012 and January 2015 to 31 December 2015. Those two periods amount to a total of about 100 hit events at Earth and a large amount of misses. Considering a time period will remove any bias in selecting events and the event set will represent a sample set that will not be biased by user selection. Lastly, we have defined the basic metrics and skill scores that the CME Arrival and Impact working team will focus on.

Explaining the Generalized Cross-Validation on Linear Models

Cross-Validation is a model validation method widely used by the scientific community. The Generalized Cross-Validation (GCV) is an invariant version of the usual Cross-Validation method. This generalization was obtained using the non usual theory of circulant complex matrices. In this work we intend to give a clear and complete exposition concerning the linear algebra assumptions required by the theory. The aim was to make this text accessible to a wide audience of statisticians and non-statisticians who use the Cross-Validation method in their research activities. It is also intended to supply the absence of a basic reference on this topic in the literature.

Multi-scale finite element model validation method of cable-stayed bridge based on the support vector regression

In this paper, the multi-scale finite element model (FEM) of a composite cable-stayed bridge, Guanhe Bridge, was established based on the Arlequin method firstly. Then a two-step multi-scale FE mod...

Multi-scale finite element model validation method of cable-stayed bridge based on the support vector regression

Multi-scale finite element model validation method of cable-stayed bridge based on the support vector regression

The Analysis of Dynamic O/D Adjustment for Bicycle Traffic Demand Estimation with AIMSUN Simulation Model: A Case Study of Nakhon Sawan Municipality in Thailand

Aim: This research aimed to present the process of estimating bicycle traffic demand in order to design bike routes that meet the daily transportation needs of the people in Nakhon Sawan Municipality. Methods: The primary and secondary traffic data were collected to develop a virtual traffic simulation model with the use of the AIMSUN simulation software. The model validation method was carried out to adjust the origin and destination survey data (O/D matrix) by running dynamic O/D adjustment. The 99 replication scenarios were statistically examined and assessed using the goodness-of-fit test. The 9 measures, which were examined, included: 1) Root Mean Square Error (RMSE), 2) Root Mean Square Percentage Error (RMSPE%), 3) Mean Absolute Deviation (MAD), 4) Mean Bias Error (MBE), 5) Mean Percentage Error (MPE%), 6) Mean Absolute Percentage Error (MAPE%), 7) Coefficient of Determination (R2), 8) GEH Statistic (GEH), and 9) Thiel’s U Statistic (Theil’s U). Results: The resulting statistical values were used to determine the acceptable ranges according to the acceptable indicators of each factor. Conclusion: It was found that there were only 8 scenarios that met the evaluation criteria. The selection and ranking process was consequently carried out using the multi-factor scoring method, which could eliminate errors that might arise from applying only one goodness-of-fit test measure.

PENGEMBANGAN MODEL PEMBELAJARAN BERBASIS TUTORIAL BAGI MAHASISWA TEKNIK MESIN STT PLN

Tutorial method is a way of delivering learning materials that have been developed in the form of modules for students to learn independently. Students can consult about the problems and progress encountered periodically. Many limitations teacher, in choosing the right method of teaching cause lecturer difficult to realize the demands of the students in this teach. Acceleration we can see at universities around us, still very much a lecturer who uses teaching methods in a conventional way, such as lectures, notes, summarizes and etc. Research aims to determine the effectiveness of the use of the tutorial method of teaching in the department of mechanical engineering. This study uses research and development (Research and Development), the stage of preliminary studies carried out by applying a qualitative descriptive approach. The development phase is done by trial limited to the conventional model of teaching faculty by applying experimental methods (Single One Shot Case Study) After no improvement on the limited test then continued with a broader test of the experimental method in the form of groups (one group pretest-posttest). The latter is a tutorial teaching model validation methods eksprimen. The result of research shows that tutorial teaching methods more effective than the old teaching methods, both on the speed aspect of students' understanding of the lecture material, creativity, and student results.