Model Validation Strategy Research Articles

Covariance‐based online validation of video tracking

A novel approach is proposed for online evaluation of video tracking without ground-truth data. The temporal evolution of the covariance features is exploited to detect the stability of the tracker output over time. A model validation strategy performs such detection without learning the failure cases of the tracker under evaluation. Then, the tracker performance is estimated by a finite state machine determining whether the tracker is on-target (successful) or not (unsuccessful). The experimental results over a heterogeneous dataset show that the proposed approach outperforms related state-of-the-art approaches in terms of performance and computational cost.

Animal models of gastrointestinal and liver diseases. Animal models of necrotizing enterocolitis: pathophysiology, translational relevance, and challenges.

Necrotizing enterocolitis is the leading cause of morbidity and mortality from gastrointestinal disease in premature infants and is characterized by initial feeding intolerance and abdominal distention followed by the rapid progression to coagulation necrosis of the intestine and death in many cases. Although the risk factors for NEC development remain well accepted, namely premature birth and formula feeding, the underlying mechanisms remain incompletely understood. Current thinking indicates that NEC develops in response to an abnormal interaction between the mucosal immune system of the premature host and an abnormal indigenous microflora, leading to an exaggerated mucosal inflammatory response and impaired mesenteric perfusion. In seeking to understand the molecular and cellular events leading to NEC, various animal models have been developed. However, the large number and variability between the available animal models and the unique characteristics of each has raised important questions regarding the validity of particular models for NEC research. In an attempt to provide some guidance to the growing community of NEC researchers, we now seek to review the key features of the major NEC models that have been developed in mammalian and nonmammalian species and to assess the advantages, disadvantage, challenges and major scientific discoveries yielded by each. A strategy for model validation is proposed, the principal models are compared, and future directions and challenges within the field of NEC research are explored.

Modeling strategies of membrane contactors for post-combustion carbon capture: A critical comparative study

Membrane contactors are considered as a promising process for intensification purposes of gas–liquid absorption units. CO2 post-combustion capture is one of the currently intensively investigated applications and experiments are most often performed on lab scale hollow fiber modules with a monoethanolamine (MEA) aqueous solution, usually in a large excess, as chemical solvent. Different mathematical models, showing a broad range of intrinsic complexity, have been already proposed in order to simulate or predict the separation performances of membrane contactors for this application. Unfortunately, no systematic comparison of the different modeling approaches has been performed yet. This study addresses this issue through a series of experiments performed on the two main types of hollow fiber membrane contactors (microporous PTFE and dense PMP skin composite membrane) under different sets of operating conditions. Four different types of models (constant overall mass transfer coefficient, 1D resistance in series, 1D and 2D convection–diffusion models) have been compared with the membrane mass transfer coefficient as the only adjustable parameter. It is shown that the different models lead to comparable predictions of the experimental results, with slightly similar membrane mass transfer coefficient values. This result addresses key questions in terms of strategy for model validation and with regard to the current trend of increasing model complexity. Interestingly, a different situation holds when MEA is significantly converted at the liquid outlet: in that case, a 1D model approach is required (variable mass transfer coefficient), and leads to results which are comparable to 2D models. Guidelines for a relevant model comparison strategy are finally proposed.

Derivation of biophysical variables from Earth observation data: validation and statistical measures

Evaluation is an essential step of model development. However, there is a missing definition of appropriate validation strategies, needed to guarantee reproducibility and generalizability of modeling results. Also, there is a lack of a generally agreed set of 'optimal' statistical measure(s) to assess model accuracy. The objective of the present study is to provide for remote sensing practitioners (i.e., non-statisticians) guidance for model validation strategies and to propose an optimal set of statistical measures for the quantitative assessment of model performance in the context of vegetation biophysical variable retrieval from Earth observation (EO) data. For these purposes, main terms and concepts were reviewed. Then, validation strategies were tested on a polynomial regression model and discussed. Moreover, a literature review was carried out, summarizing the statistical measures used to evaluate model performances. Supported by some exemplary datasets, these measures were calculated and their meanings discussed in view of several model validation criteria. From the results, we recommend to further exploit cross-validation and bootstrapping strategies to guarantee the development/validation of reliable models. An 'optimal' statistic set is suggested, including root mean square error (RMSE), coefficient of determination (R 2 ), slope and intercept of Theil-Sen regression, relative RMSE, and Nash-Sutcliffe efficiency index. A wide acceptance and use of these statistics should enable a better intercomparison of scientific results, urgently needed in times of increasing model development activities that are carried out with respect to upcoming EO missions.

A Simple State-Based Prognostic Model for Railway Turnout Systems

The importance of railway transportation has been increasing in the world. Considering the current and future estimates of high cargo and passenger transportation volume in railways, prevention or reduction of delays due to any failure is becoming ever more crucial. Railway turnout systems are one of the most critical pieces of equipment in railway infrastructure. When incipient failures occur, they mostly progress slowly from the fault-free to the failure state. Although studies focusing on the identification of possible failures in railway turnout systems exist in literature, neither the detection nor forecasting of failure progression has been reported. This paper presents a simple state-based prognostic (SSBP) method that aims to detect and forecast failure progression in electromechanical systems. The method is compared with Hidden-Markov-Model-based methods on real data collected from a railway turnout system. Obtaining statistically sufficient failure progression samples is difficult, considering that the natural progression of failures in electromechanical systems may take years. In addition, validating the classification model is difficult when the degradation is not observable. Data collection and model validation strategies for failure progression are also presented.

Advanced simulation environment for clean power production in IGCC plants

Abstract Oxygen-blown biomass integrated gasification combined cycle (IGCC) plants are one of the most promising options for clean energy generation with CO 2 abatement potential. However, the integrated nature of IGCC leads to difficult design problems. In this study, we present an advanced simulation environment for the preliminary design and retrofit of IGCC plants. We describe the modelling approach, the model validation strategy and the plant behaviour, as determined by sensitivity analyses. The simulation environment uses Pareto curves to examine various co-gasification and co-production case studies in terms of technical, economic and environmental performance. It serves as a decision support tool in the design stage, which can be used to explore ways to improve plant performance and to analyse the influence of raw materials and the unit’s operational parameters. The test and validation results are discussed.

Evaluation of the therapeutic potential of carbonic anhydrase inhibitors in two animal models of dystrophin deficient muscular dystrophy

Duchenne Muscular Dystrophy is an inherited muscle degeneration disease for which there is still no efficient treatment. However, compounds active on the disease may already exist among approved drugs but are difficult to identify in the absence of cellular models. We used the Caenorhabditis elegans animal model to screen a collection of 1000 already approved compounds. Two of the most active hits obtained were methazolamide and dichlorphenamide, carbonic anhydrase inhibitors widely used in human therapy. In C. elegans, these drugs were shown to interact with CAH-4, a putative carbonic anhydrase. The therapeutic efficacy of these compounds was further validated in long-term experiments on mdx mice, the mouse model of Duchenne Muscular Dystrophy. Mice were treated for 120 days with food containing methazolamide or dichlorphenamide at two doses each. Musculus tibialis anterior and diaphragm muscles were histologically analyzed and isometric muscle force was measured in M. extensor digitorum longus. Both substances increased the tetanic muscle force in the treated M. extensor digitorum longus muscle group, dichlorphenamide increased the force significantly by 30%, but both drugs failed to increase resistance of muscle fibres to eccentric contractions. Histological analysis revealed a reduction of centrally nucleated fibers in M. tibialis anterior and diaphragm in the treated groups. These studies further demonstrated that a C. elegans-based screen coupled with a mouse model validation strategy can lead to the identification of potential pharmacological agents for rare diseases.

A regression-based equivalence test for model validation: shifting the burden of proof

Model validation is often realized as a test of how well model predictions match a set of independent observations. One would think that the burden of proof should rest with the model, to force it to show that it can make accurate predictions. Further, one would think that increasing the sample size ought to increase the model's ability to demonstrate its utility. Traditional statistical tools are inappropriate for this because they default to the case that the model and the data are no different, and their ability to detect differences increases with the sample size. These traditional tools are optimized to detect differences, rather than similarities. We present an alternative strategy for model validation that is based on regression and statistical tests of equivalence. Equivalence tests reverse the usual null hypothesis: they posit that the populations being compared are different and use the data to prove otherwise. In this sense, equivalence tests are lumping tests, whereas the traditional statistical tests are splitting tests. To date, model validation with equivalence tests has focused on comparisons of means. Our proposed test checks not only for similarity of means, but also for similarity between individual predictions and observations. The strategy is demonstrated using three case studies that differ in their modeling objectives, and for varied sample sizes. The proposed strategy provides a formal means of model validation that is superior to traditional statistical tests in each case.

Kinetic simulation of signal transduction system in hippocampal long-term potentiation with dynamic modeling of protein phosphatase 2A

We modeled and analyzed a signal transduction system of long-term potentiation (LTP) in hippocampal post-synapse. Bhalla and Iyengar [Science 283(1999) 381] have developed a hippocampal LTP model. In the conventional model, the concentration of protein phosphatase 2A (PP2A) was fixed. However, it was reported that dynamic inactivation of PP2A was essential for LTP [J. Neurochem. 74 (2000) 807]. We introduced a dynamic modeling of PP2A; inactivation (phosphorylation) of PP2A by calcium/calmodulin-dependent protein kinase II (CaMKII) in the presence of calcium/calmodulin, self-activation (autodephosphorylation) of PP2A, and inactivation (dephosphorylation) of CaMKII by PP2A. This model includes complex feedback loops; both CaMKII and PP2A are autoactivated, while they inactivate each other. Moreover, we proposed an analysis strategy for model validation by applying the results of sensitivity analysis. In our system, calcineurin (CaN) played an essential role, rather than the activation of protein kinase C (PKC) as documented in the conventional model. From results of the analysis of our model, we found the following robustness as characteristics of bistability in our model: (1) PP2A reactions against calcium ion (Ca 2+) perturbation; (2) PP2A inactivation against PP2A increase; (3) protein phosphatase 1 (PP1) activation against PF2A increase; and (4) PP2A reactions against PP2A initial concentration. These properties facilitated LTP induction in our system. We showed that another mechanism could introduce bistable behavior by adding dynamic reactions of PP2A.

The spatial variability of soil nitrates in arable and pasture landscapes: implications for the development of geographical information system models of nitrate leaching

Abstract. In response to the European Community Nitrate Directive (91/676) a catchment scale Geographical Information System (GIS) model of nitrate leaching has been developed to map nitrate vulnerability and predict average weekly fluxes of nitrate from agricultural land units to surface water. This paper presents a pilot study which investigated the spatial variability of soil nitrates in order to: (1) define an appropriate pixel size for modelling N leaching; (2) quantify the within‐unit variability of soil nitrate concentrations for pasture and arable fields; and (3) assist in the design of an efficient sampling strategy for estimating mean nitrate concentrations. Soil samples, taken from two 800 m transects in early September 1994, were analysed for water soluble nitrate. The arable soils had a mean nitrate‐nitrogen concentration of 0.693 μg/g (S.E. 0.054 μg/g) and the pasture soils had a higher mean nitrate‐nitrogen concentration of 0.86 μg/g (S.E. 0.085 μg/g). Spatial variability was investigated using variograms. The pasture data had a weak spatial relationship, whereas the arable data exhibited a strong spatial relationship which fitted a spherical variogram model (r2 0.87), with a range of 40 m. A pixel size of 40 m is suggested for nitrate modelling within the GIS based on the arable variogram and an improved sampling strategy for model validation is suggested, involving bulking sub‐samples over a 40 m grid for estimating mean nitrate concentrations in combined land use and soil units.

Testing procedures for spatially distributed flow models

Methods for testing, verifying, and validating predictive models of variably saturated groundwater flow are discussed. Specific procedures are introduced for measuring model complexity, assessing model consistency, and testing model validity. The discussion addresses numerical formulation, verification of internal consistency, benchmarking, groundtruth testing, performance measures, parameter estimation, hypothesis testing, and probabilistic induction. Verification of models includes tests of internal consistency and accuracy, like mass conservation and sensitivity to mesh size. Verification of codes also involves comparing results from the numerical model to analytical solutions, which are, however, limited in scope, and comparison with other numerical codes or ‘benchmarking’. These aspects are illustrated using available three-dimensional codes developed by the authors. Recognizing the diversity of spatially distributed modeling approaches, we also propose measures of model complexity and of the amount of information inherent in model predictions. One of these measures is the spatial degree of freedom, a function of material and boundary heterogeneities in the model. Another one is the quantity of information or entropy, which depends also on precision. Several aspects of ‘groundtruth’ model validation using data from laboratory and field tests are discussed. Logical inference is used to distinguish model validation from refutation. Recognizing that full validation is not possible in practice, we formulate performance criteria to define the ‘degree of validation’. Concepts and methods based on inductive calculus, Bayesian hypothesis testing, and maximum likelihood, are analyzed in some detail as alternative model validation strategies. Several examples of model testing are also discussed.