Model Validation Method Research Articles

Progress in space-borne studies of permafrost for climate science: Towards a multi-ECV approach

Remotely-sensed climate data records (CDRs) provide a basis for spatially distributed global climate model (GCM) inputs and validation methods. GCMs can take advantage of land surface models (LSMs), which aim to resolve surface energy, water and carbon budgets and hence these LSMs present important boundary conditions at the land-atmosphere interface. Pertinently, satellite data assimilation approaches are essential for improved land surface modelling for northern high latitudes ecosystems where permafrost degradation is reported to be ongoing. Permafrost, however, is an Essential Climate Variable (ECV) that cannot directly be monitored from space.Here, we advocate that CDRs, such as those compiled under the European Space Agency (ESA) Climate Change Initiative (CCI) programme, may be used in combination with permafrost models to improve our understanding of permafrost extent and degradation in a changing climate system. We describe the current types of remotely-sensed surface feature products that are widely used as indicators for permafrost related features. Furthermore, we highlight issues of using these site-specific permafrost proxies related to spatial scale, as well as the uncertainties in establishing present-day permafrost extent itself.Our assessment of the key ECVs that impact on permafrost, demonstrates how models that incorporate EO CDRs have the potential to boost our knowledge of permafrost conditions through better parametrisation of the thermal regime of permafrost soils.

행정업무시스템의 생산성 및 효과 분석: 나라장터 중심으로

The evaluation and analysis method of information system (IS) is studied from the system perspective, the user perspective, and the management viewpoint. The detailed analysis method performs qualitative evaluation by user questionnaire or expert opinion. In this study, Measures the productivity and the effect of building administrative information systems. In the previous study, qualitative productivity and universal effect indicators were used, but in this study, quantitative productivity indicators and indicators specific to administrative complaints were selected. KONEPS, an administrative service system, used electronic contract records and information recorded in the intermediate process. The information was converted into the number of days, and the productivity based on the input manpower was calculated. The effect analysis analyzed the questionnaire related to civil affairs, which is the goal of the administrative work system. Each factor was divided into reflective structural variable and formal structural variable, and internal consistency and multi-collinearity were diagnosed. In order to verify the model, the influence of the work was set as a hypothesis, the reliability was verified according to the descriptive statistics method, the influence was measured through the regression analysis, and the model was analyzed by the multiple regression model path coefficient. Model validation methods are Chi-square (df, p), RMR, GFI, AGFI, NFI, CFI and GFI as indicators according to CFA.

Strength and Viscosity Effects on Perturbed Shock Front Stability in Metals

Computational modeling and experimental measurements on metal samples subject to a laser-driven, ablative Richtmyer-Meshkov instability showed differences between viscosity and strength effects. In particular, numerical and analytical solutions, coupled with measurements of fed-through perturbations, generated by perturbed shock fronts onto initially flat surfaces, show promise as a validation method for models of deviatoric response in the postshocked material. Analysis shows that measurements of shock perturbation amplitudes at low sample thickness-to-wavelength ratios are not enough to differentiate between strength and viscosity effects, but that surface displacement data of the fed-through perturbations appears to resolve the ambiguity. Additionally, analytical and numerical results show shock front perturbation evolution dependence on initial perturbation amplitude and wavelength is significantly different in viscous and materials with strength, suggesting simple experimental geometry changes should provide data supporting one model or the other.

Influence of varying the input parameters on the results of model validation

This article presents an evaluation of the model validation method that was provided as the output of the European research project DynoTRAIN and implemented in the recently revised European standard EN 14363. The input parameters of the validation method, namely the section length, number, and selection of sections as well as the selected parameters of the simulation models are varied. The evaluation shows that a single section that provides a large deviation between simulation and measurement can, in rare cases, influence the results of the overall validation. Nevertheless, the investigations demonstrate a good robustness, as the final validation result is very rarely influenced by the variation of sections selected for validation, by the use of a higher number of sections than the minimum of 12, or by longer sections than that specified for on-track tests in accordance with EN 14363. The validation methodology is also able to recognize the errors in vehicle model parameters, if the errors have a relevant influence on the behaviour of the running dynamics of the evaluated vehicle.

Analysis of a Compartmental Model of Endogenous Immunoglobulin G Metabolism with Application to Multiple Myeloma.

Immunoglobulin G (IgG) metabolism has received much attention in the literature for two reasons: (i) IgG homeostasis is regulated by the neonatal Fc receptor (FcRn), by a pH-dependent and saturable recycling process, which presents an interesting biological system; (ii) the IgG-FcRn interaction may be exploitable as a means for extending the plasma half-life of therapeutic monoclonal antibodies, which are primarily IgG-based. A less-studied problem is the importance of endogenous IgG metabolism in IgG multiple myeloma. In multiple myeloma, quantification of serum monoclonal immunoglobulin plays an important role in diagnosis, monitoring and response assessment. In order to investigate the dynamics of IgG in this setting, a mathematical model characterizing the metabolism of endogenous IgG in humans is required. A number of authors have proposed a two-compartment nonlinear model of IgG metabolism in which saturable recycling is described using Michaelis–Menten kinetics; however it may be difficult to estimate the model parameters from the limited experimental data that are available. The purpose of this study is to analyse the model alongside the available data from experiments in humans and estimate the model parameters. In order to achieve this aim we linearize the model and use several methods of model and parameter validation: stability analysis, structural identifiability analysis, and sensitivity analysis based on traditional sensitivity functions and generalized sensitivity functions. We find that all model parameters are identifiable, structurally and taking into account parameter correlations, when several types of model output are used for parameter estimation. Based on these analyses we estimate parameter values from the limited available data and compare them with previously published parameter values. Finally we show how the model can be applied in future studies of treatment effectiveness in IgG multiple myeloma with simulations of serum monoclonal IgG responses during treatment.

Reliability-Based Design Optimization Using Confidence-Based Model Validation for Insufficient Experimental Data

The conventional reliability-based design optimization (RBDO) methods assume that a simulation model is able to represent the real physics accurately. However, this assumption may not always hold as the simulation model could be biased. Accordingly, designed product based on the conventional RBDO optimum may either not satisfy the target reliability or be overly conservative design. Therefore, simulation model validation using output experimental data, which corrects model bias, should be integrated in the RBDO process. With particular focus on RBDO, the model validation needs to account for the uncertainty induced by insufficient experimental data as well as the inherent variability of the products. In this paper, a confidence-based model validation method that captures the variability and the uncertainty, and that corrects model bias at a user-specified target confidence level, has been developed. The developed model validation helps RBDO to obtain a conservative RBDO optimum design at the target confidence level. The RBDO with model validation may have a convergence issue because the feasible domain changes as the design moves (i.e., a moving-target problem). To resolve the issue, a practical optimization procedure is proposed. Furthermore, the efficiency is achieved by carrying out deterministic design optimization (DDO) and RBDO without model validation, followed by RBDO with confidence-based model validation. Finally, we demonstrate that the proposed RBDO approach can achieve a conservative and practical optimum design given a limited number of experimental data.

Validación de modelos con sesgo proporcional

Objetivo. En este trabajo se presentan extensiones al método estadístico de Freese para validar modelos con sesgo proporcional (SP) en sus predicciones y se ilustra el método con datos provenientes de un modelo de simulación de crecimiento de praderas. Materiales y métodos. Las extensiones para validar un modelo con SP fueron: el error máximo anticipado para el planteamiento original, la prueba de hipótesis y error máximo anticipado para el planteamiento alternativo, y el intervalo de confianza para un cuantil de la distribución de los errores. Resultados. El modelo evaluado presentó SP, una vez removido y con un nivel de confianza del 95% la magnitud del error no sobrepasa 1225.564 kg ha-1. Por lo que el modelo validado podría usarse para predecir el crecimiento de praderas, sin embargo, requerirá un ajuste en su estructura con base a la presencia de SP. Conclusiones. Las extensiones presentadas para validar modelos en presencia de SP se aplican sin que el modelo sea modificado en su estructura. El intervalo de confianza para el cuantil 1-α de la distribución de los errores una vez que se corrige el SP, permite determinar una cota superior para la magnitud del error de predicción y usarla para evaluar la evolución del modelo en predicción del sistema.

Model validation method with multivariate output based on kernel principal component analysis

Model validation method with multivariate output based on kernel principal component analysis

Effect of monitoring network design on land use regression models for estimating residential NO2 concentration

Land-use regression (LUR) models are increasingly used to estimate exposure to air pollution in urban areas. An appropriate monitoring network is an important component in the development of a robust LUR model. In this study concentrations of NO2 were simulated by a dispersion model at ‘virtual’ monitoring sites in 54 network designs of varying numbers and types of site, using a 25 km2 area in Edinburgh, UK, as an example location. Separate LUR models were developed for each network. The LUR models were then used to estimate NO2 concentration at all residential addresses, which were evaluated against the dispersion-modelled concentration at these addresses. The improvement in predictive capability of the LUR models was insignificant above ∼30 monitoring sites, although more sites tended to yield more precise LUR models. Monitoring networks containing sites located within highly populated areas better estimated NO2 concentrations across all residential locations. LUR models constructed from networks containing more roadside sites better characterised the high end of residential NO2 concentrations but had increased errors when considering the whole range of concentrations. No particular composition of monitoring network resulted in good estimation simultaneously across all residential NO2 concentration and of the highest NO2 levels. This evaluation with dispersion modelling has shown that previous LUR model validation methods may have been optimistic in their assessment of the model's predictive performance at residential locations.

A Visual Approach for Data-Intensive Workflow Validation

This paper presents a workflow validation method for data-intensive graphical workflow models using real-time workflow tracing mode on data-intensiveworkflow designer. In order to model and validate workflows, we try to divide as modes have editable mode and tracingmode on data-intensiveworkflow designer. We could design data-intensive workflow using drag and drop in editable-mode, otherwise we could not design but view and trace workflow model in tracing mode. We would like to focus on tracing-mode for workflow validation, and describe how to use workflow tracing on data-intensive workflow model designer. Especially, it is support data centered operation about control logics and exchange variables on workflow runtime for workflow tracing.

An Adaptive Goal-Based Model for Autonomous Multi-Robot Using HARMS and NuSMV

In a dynamic environment autonomous robots often encounter unexpected situations that the robots have to deal with in order to continue proceeding their mission. We propose an adaptive goal-based model that allows cyber-physical systems (CPS) to update their environmental model and helps them analyze for attainment of their goals from current state using the updated environmental model and its capabilities. Information exchange approach utilizes Human-Agent-Robot-Machine-Sensor (HARMS) model to exchange messages between CPS. Model validation method uses NuSMV, which is one of Model Checking tools, to check whether the system can continue its mission toward the goal in the given environment. We explain a practical set up of the model in a situation in which homogeneous robots that has the same capability work in the same environment.

Generic data structure and validation methodology for simulation of manufacturing systems

In recent years, the analysis and evaluation of manufacturing systems’ behaviour and their performance became essential. Discrete event simulation (DES) as a digital enterprise technology is an effective tool both in production related decision support processes and in structure or performance analysis of manufacturing systems. Building a DES model of a manufacturing system is a difficult task and requires special competence. Reducing the efforts spent on draft simulation is the aim of an ongoing research which is presented in the paper. The developed framework simplifies and accelerates the process of model building. A production oriented implementation of the ANSI/ISA-95 standard was used by the proposed modelling methodology to define a generic data structure supporting the creation of models without specific knowledge related to the simulation software applied. The data structure enables the development and application of proprietary simulation engines tailored for specific problems. On the other hand, numerous objective tests containing simulation model validation methods are also announced through a well-described example problem. By this method, several validating key performance indicators (KPIs) were tested, and it was indirectly determined whether the simulation model adequately represents the behaviour of the real system. The automatic simulation model generation in our approach enables using different simulation tools and presented through the examples of both commercial and self-developed software.

Numerical simulation and experimental verification of heat transfer from a finned housing of an electric motor

Engineering analyses based on the results of numerical simulation enable shortening the duration and reducing the costs of the development and tests of equipment models or prototypes. As numerical modeling methods have uncertainties, experimental verification is required. In the literature, there are many examples of application of numerical modelling to the thermal processes occurring in electric motors but the experimental methods of model validation are not so well described.In this work, numerical and experimental investigations of heat conduction in the finned housing of an electric motor, and the comparison between their results are presented. 3D geometrical model of the motor rated 7.5kW was imported to COMSOL Multiphysics software package. After defining thermophysical properties of the housing material and heat flows generated in the motor, and setting boundary conditions, a numerical model of heat conduction in the motor housing was obtained. The model allowed to calculate steady-state temperature distribution in the housing cross section. In order to validate the model, investigations of the temperature distribution were carried out using FLIR SC7600 thermovision camera. Steady-state temperature distribution on the surface of housing of the motor run in idling conditions was recorded by thermography. The obtained thermograms were found to be in satisfactory quantitative agreement with the simulated temperature distribution as the real values of temperature differences measured across the housing cross section deviated not more than 10% from the calculated ones. The research confirmed that a thermovision camera is an efficient experimental tool for the validation of numerical models of the heat transfer phenomena in the finned housing of an electric motor.

Study on the synthetic scavenging model validation method of opposed-piston two-stroke diesel engine

The scavenging process of two stroke engines is a complex phenomenon. In two stroke engines, each outward stroke of the piston is a power stroke. This means the operation of clearing the burned gas in cylinder and obtaining fresh air for next cycle must accomplish at same time. The combined intake and exhaust process in two stroke engine is called scavenging. Nowadays, a synthetic scavenging model is widely used to describe the scavenging process of two stroke engines. It represents the time evolution of the in-cylinder residual gas rate and the exhaust residual gas rate. An opposed two stroke diesel engine’s (OP2S) synthetic scavenging profile is calculated through the changes of gas compositions in-cylinder and exhaust pipe by CFD simulation. To validate the accuracy of this profile, a validation method based on tracer gas technique is proposed. First, scavenging parameters including scavenging efficiency and trapping efficiency are calculated by 1D software using the synthetic scavenging profile as a vital boundary condition. The results reveal that the delivery ratio of the OP2S diesel engine should below 140%. Second, use tracer gas technique to measure the scavenging parameters of OP2S diesel engine. In the experimental process, we find the OP2S diesel engine operates steady only if the delivery ratio is higher 50%. The scavenging parameters from 1D simulation and experiment match well which means the synthetic scavenging model received by CFD simulation performs well and validating its effectiveness through tracer gas technique is available.

Fall Risk Assessment Through Automatic Combination of Clinical Fall Risk Factors and Body-Worn Sensor Data.

Falls are the leading global cause of accidental death and disability in older adults and are the most common cause of injury and hospitalization. Accurate, early identification of patients at risk of falling, could lead to timely intervention and a reduction in the incidence of fall-related injury and associated costs. We report a statistical method for fall risk assessment using standard clinical fall risk factors (N = 748). We also report a means of improving this method by automatically combining it, with a fall risk assessment algorithm based on inertial sensor data and the timed-up-and-go test. Furthermore, we provide validation data on the sensor-based fall risk assessment method using a statistically independent dataset. Results obtained using cross-validation on a sample of 292 community dwelling older adults suggest that a combined clinical and sensor-based approach yields a classification accuracy of 76.0%, compared to either 73.6% for sensor-based assessment alone, or 68.8% for clinical risk factors alone. Increasing the cohort size by adding an additional 130 subjects from a separate recruitment wave (N = 422), and applying the same model building and validation method, resulted in a decrease in classification performance (68.5% for combined classifier, 66.8% for sensor data alone, and 58.5% for clinical data alone). This suggests that heterogeneity between cohorts may be a major challenge when attempting to develop fall risk assessment algorithms which generalize well. Independent validation of the sensor-based fall risk assessment algorithm on an independent cohort of 22 community dwelling older adults yielded a classification accuracy of 72.7%. Results suggest that the present method compares well to previously reported sensor-based fall risk assessment methods in assessing falls risk. Implementation of objective fall risk assessment methods on a large scale has the potential to improve quality of care and lead to a reduction in associated hospital costs, due to fewer admissions and reduced injuries due to falling.

A multiscale finite element model validation method of composite cable-stayed bridge based on Probability Box theory

Modeling and simulation are routinely implemented to predict the behavior of complex structures. These tools powerfully unite theoretical foundations, numerical models and experimental data which include associated uncertainties and errors. A new methodology for multi-scale finite element (FE) model validation is proposed in this paper. The method is based on two-step updating method, a novel approach to obtain coupling parameters in the gluing sub-regions of a multi-scale FE model, and upon Probability Box (P-box) theory that can provide a lower and upper bound for the purpose of quantifying and transmitting the uncertainty of structural parameters. The structural health monitoring data of Guanhe Bridge, a composite cable-stayed bridge with large span, and Monte Carlo simulation were used to verify the proposed method. The results show satisfactory accuracy, as the overlap ratio index of each modal frequency is over 89% without the average absolute value of relative errors, and the CDF of normal distribution has a good coincidence with measured frequencies of Guanhe Bridge. The validated multiscale FE model may be further used in structural damage prognosis and safety prognosis.

Evaluation of Scat Deposition Transects versus Radio Telemetry for Developing a Species Distribution Model for a Rare Desert Carnivore, the Kit Fox.

Development and evaluation of noninvasive methods for monitoring species distribution and abundance is a growing area of ecological research. While noninvasive methods have the advantage of reduced risk of negative factors associated with capture, comparisons to methods using more traditional invasive sampling is lacking. Historically kit foxes (Vulpes macrotis) occupied the desert and semi-arid regions of southwestern North America. Once the most abundant carnivore in the Great Basin Desert of Utah, the species is now considered rare. In recent decades, attempts have been made to model the environmental variables influencing kit fox distribution. Using noninvasive scat deposition surveys for determination of kit fox presence, we modeled resource selection functions to predict kit fox distribution using three popular techniques (Maxent, fixed-effects, and mixed-effects generalized linear models) and compared these with similar models developed from invasive sampling (telemetry locations from radio-collared foxes). Resource selection functions were developed using a combination of landscape variables including elevation, slope, aspect, vegetation height, and soil type. All models were tested against subsequent scat collections as a method of model validation. We demonstrate the importance of comparing multiple model types for development of resource selection functions used to predict a species distribution, and evaluating the importance of environmental variables on species distribution. All models we examined showed a large effect of elevation on kit fox presence, followed by slope and vegetation height. However, the invasive sampling method (i.e., radio-telemetry) appeared to be better at determining resource selection, and therefore may be more robust in predicting kit fox distribution. In contrast, the distribution maps created from the noninvasive sampling (i.e., scat transects) were significantly different than the invasive method, thus scat transects may be appropriate when used in an occupancy framework to predict species distribution. We concluded that while scat deposition transects may be useful for monitoring kit fox abundance and possibly occupancy, they do not appear to be appropriate for determining resource selection. On our study area, scat transects were biased to roadways, while data collected using radio-telemetry was dictated by movements of the kit foxes themselves. We recommend that future studies applying noninvasive scat sampling should consider a more robust random sampling design across the landscape (e.g., random transects or more complete road coverage) that would then provide a more accurate and unbiased depiction of resource selection useful to predict kit fox distribution.

PyEMMA 2: A Software Package for Estimation, Validation, and Analysis of Markov Models.

Markov (state) models (MSMs) and related models of molecular kinetics have recently received a surge of interest as they can systematically reconcile simulation data from either a few long or many short simulations and allow us to analyze the essential metastable structures, thermodynamics, and kinetics of the molecular system under investigation. However, the estimation, validation, and analysis of such models is far from trivial and involves sophisticated and often numerically sensitive methods. In this work we present the open-source Python package PyEMMA ( http://pyemma.org ) that provides accurate and efficient algorithms for kinetic model construction. PyEMMA can read all common molecular dynamics data formats, helps in the selection of input features, provides easy access to dimension reduction algorithms such as principal component analysis (PCA) and time-lagged independent component analysis (TICA) and clustering algorithms such as k-means, and contains estimators for MSMs, hidden Markov models, and several other models. Systematic model validation and error calculation methods are provided. PyEMMA offers a wealth of analysis functions such that the user can conveniently compute molecular observables of interest. We have derived a systematic and accurate way to coarse-grain MSMs to few states and to illustrate the structures of the metastable states of the system. Plotting functions to produce a manuscript-ready presentation of the results are available. In this work, we demonstrate the features of the software and show new methodological concepts and results produced by PyEMMA.

Reporting and Methodology of Multivariable Analyses in Prognostic Observational Studies Published in 4 Anesthesiology Journals

Prognostic research studies in anesthesiology aim to identify risk factors for an outcome (explanatory studies) or calculate the risk of this outcome on the basis of patients' risk factors (predictive studies). Multivariable models express the relationship between predictors and an outcome and are used in both explanatory and predictive studies. Model development demands a strict methodology and a clear reporting to assess its reliability. In this methodological descriptive review, we critically assessed the reporting and methodology of multivariable analysis used in observational prognostic studies published in anesthesiology journals. A systematic search was conducted on Medline through Web of Knowledge, PubMed, and journal websites to identify observational prognostic studies with multivariable analysis published in Anesthesiology, Anesthesia & Analgesia, British Journal of Anaesthesia, and Anaesthesia in 2010 and 2011. Data were extracted by 2 independent readers. First, studies were analyzed with respect to reporting of outcomes, design, size, methods of analysis, model performance (discrimination and calibration), model validation, clinical usefulness, and STROBE (i.e., Strengthening the Reporting of Observational Studies in Epidemiology) checklist. A reporting rate was calculated on the basis of 21 items of the aforementioned points. Second, they were analyzed with respect to some predefined methodological points. Eighty-six studies were included: 87.2% were explanatory and 80.2% investigated a postoperative event. The reporting was fairly good, with a median reporting rate of 79% (75% in explanatory studies and 100% in predictive studies). Six items had a reporting rate <36% (i.e., the 25th percentile), with some of them not identified in the STROBE checklist: blinded evaluation of the outcome (11.9%), reason for sample size (15.1%), handling of missing data (36.0%), assessment of colinearity (17.4%), assessment of interactions (13.9%), and calibration (34.9%). When reported, a few methodological shortcomings were observed, both in explanatory and predictive studies, such as an insufficient number of events of the outcome (44.6%), exclusion of cases with missing data (93.6%), or categorization of continuous variables (65.1%.). The reporting of multivariable analysis was fairly good and could be further improved by checking reporting guidelines and EQUATOR Network website. Limiting the number of candidate variables, including cases with missing data, and not arbitrarily categorizing continuous variables should be encouraged.

Ecological approaches in veterinary epidemiology: mapping the risk of bat-borne rabies using vegetation indices and night-time light satellite imagery.

Rabies remains a disease of significant public health concern. In the Americas, bats are an important source of rabies for pets, livestock, and humans. For effective rabies control and prevention, identifying potential areas for disease occurrence is critical to guide future research, inform public health policies, and design interventions. To anticipate zoonotic infectious diseases distribution at coarse scale, veterinary epidemiology needs to advance via exploring current geographic ecology tools and data using a biological approach. We analyzed bat-borne rabies reports in Chile from 2002 to 2012 to establish associations between rabies occurrence and environmental factors to generate an ecological niche model (ENM). The main rabies reservoir in Chile is the bat species Tadarida brasiliensis; we mapped 726 occurrences of rabies virus variant AgV4 in this bat species and integrated them with contemporary Normalized Difference Vegetation Index (NDVI) data from the Moderate Resolution Imaging Spectroradiometer (MODIS). The correct prediction of areas with rabies in bats and the reliable anticipation of human rabies in our study illustrate the usefulness of ENM for mapping rabies and other zoonotic pathogens. Additionally, we highlight critical issues with selection of environmental variables, methods for model validation, and consideration of sampling bias. Indeed, models with weak or incorrect validation approaches should be interpreted with caution. In conclusion, ecological niche modeling applications for mapping disease risk at coarse geographic scales have a promising future, especially with refinement and enrichment of models with additional information, such as night-time light data, which increased substantially the model’s ability to anticipate human rabies.Electronic supplementary materialThe online version of this article (doi:10.1186/s13567-015-0235-7) contains supplementary material, which is available to authorized users.