Importance Of Model Validation Research Articles

Breast Cancer Risk Assessment in the AI Era: The Importance of Model Validation in Ethnically Diverse Cohorts.

Breast Cancer Risk Assessment in the AI Era: The Importance of Model Validation in Ethnically Diverse Cohorts.

Applicability domains of neural networks for toxicity prediction

<abstract> <p>In this paper, the term "applicability domain" refers to the range of chemical compounds for which the statistical quantitative structure-activity relationship (QSAR) model can accurately predict their toxicity. This is a crucial concept in the development and practical use of these models. First, a multidisciplinary review is provided regarding the theory and practice of applicability domains in the context of toxicity problems using the classical QSAR model. Then, the advantages and improved performance of neural networks (NNs), which are the most promising machine learning algorithms, are reviewed. Within the domain of medicinal chemistry, nine different methods using NNs for toxicity prediction were compared utilizing 29 alternative artificial intelligence (AI) techniques. Similarly, seven NN-based toxicity prediction methodologies were compared to six other AI techniques within the realm of food safety, 11 NN-based methodologies were compared to 16 different AI approaches in the environmental sciences category and four specific NN-based toxicity prediction methodologies were compared to nine alternative AI techniques in the field of industrial hygiene. Within the reviewed approaches, given known toxic compound descriptors and behaviors, we observed a difficulty in being able to extrapolate and predict the effects with untested chemical compounds. Different methods can be used for unsupervised clustering, such as distance-based approaches and consensus-based decision methods. Additionally, the importance of model validation has been highlighted within a regulatory context according to the Organization for Economic Co-operation and Development (OECD) principles, to predict the toxicity of potential new drugs in medicinal chemistry, to determine the limits of detection for harmful substances in food to predict the toxicity limits of chemicals in the environment, and to predict the exposure limits to harmful substances in the workplace. Despite its importance, a thorough application of toxicity models is still restricted in the field of medicinal chemistry and is virtually overlooked in other scientific domains. Consequently, only a small proportion of the toxicity studies conducted in medicinal chemistry consider the applicability domain in their mathematical models, thereby limiting their predictive power to untested drugs. Conversely, the applicability of these models is crucial; however, this has not been sufficiently assessed in toxicity prediction or in other related areas such as food science, environmental science, and industrial hygiene. Thus, this review sheds light on the prevalent use of Neural Networks in toxicity prediction, thereby serving as a valuable resource for researchers and practitioners across these multifaceted domains that could be extended to other fields in future research.</p> </abstract>

A Continental-Scale Validation of Ecosystem Service Models

Faced with environmental degradation, governments worldwide are developing policies to safeguard ecosystem services (ES). Many ES models exist to support these policies, but they are generally poorly validated, especially at large scales, which undermines their credibility. To address this gap, we describe a study of multiple models of five ES, which we validate at an unprecedented scale against 1675 data points across sub-Saharan Africa. We find that potential ES (biophysical supply of carbon and water) are reasonably well predicted by the existing models. These potential ES models can also be used as inputs to new models for realised ES (use of charcoal, firewood, grazing resources and water), by adding information on human population density. We find that increasing model complexity can improve estimates of both potential and realised ES, suggesting that developing more detailed models of ES will be beneficial. Furthermore, in 85% of cases, human population density alone was as good or a better predictor of realised ES than ES models, suggesting that it is demand, rather than supply that is predominantly determining current patterns of ES use. Our study demonstrates the feasibility of ES model validation, even in data-deficient locations such as sub-Saharan Africa. Our work also shows the clear need for more work on the demand side of ES models, and the importance of model validation in providing a stronger base to support policies which seek to achieve sustainable development in support of human well-being.

Calibrating and Validating a Simulation Model to Identify Drivers of Urban Land Cover Change in the Baltimore, MD Metropolitan Region

We build upon much of the accumulated knowledge of the widely used SLEUTH urban land change model and offer advances. First, we use SLEUTH’s exclusion/attraction layer to identify and test different urban land cover change drivers; second, we leverage SLEUTH’s self-modification capability to incorporate a demographic model; and third, we develop a validation procedure to quantify the influence of land cover change drivers and assess uncertainty. We found that, contrary to our a priori expectations, new development is not attracted to areas serviced by existing or planned water and sewer infrastructure. However, information about where population and employment growth is likely to occur did improve model performance. These findings point to the dominant role of centrifugal forces in post-industrial cities like Baltimore, MD. We successfully developed a demographic model that allowed us to constrain the SLEUTH model forecasts and address uncertainty related to the dynamic relationship between changes in population and employment and urban land use. Finally, we emphasize the importance of model validation. In this work the validation procedure played a key role in rigorously assessing the impacts of different exclusion/attraction layers and in assessing uncertainty related to population and employment forecasts.

A posteriori analysis of analytical models for heap leaching

In this paper, we present an a posteriori validation of a heap leaching analytical model using sensitivity analysis. The model is suitable to represent the process of heap leaching of solid reactants from porous pellets and is based on a combination of the leaching kinetics at two levels: particle and heap. The validation is simple, and a sensitivity analysis can easily be performed. The simplicity of the analysis provides insight into the control and optimization of the heap leaching process without strong mathematical complexities. The model and the corresponding sensitivity analysis include the effects of heap height, particle sizes, flow rates and several operation-design variables. Using a case study, we discuss the importance of model validation based on sensitivity analysis. We also comment on several choices for optimizing the process. We had three goals in this work: a posteriori validation, highlighting the importance of model validation and providing insight into optimization choices. We can conclude that the model has good internal logic and there are no extra variables; i.e., all of the variables in the model are necessary.

Variable selection in PCA in sensory descriptive and consumer data

This paper presents a general method for identifying significant variables in multivariate models. The methodology is applied on principal component analysis (PCA) of sensory descriptive and consumer data. The method is based on uncertainty estimates from cross-validation/jack-knifing, where the importance of model validation is emphasised. Student's t-tests based on the loadings and their estimated standard uncertainties are used to calculate significance on each variable for each component. Two data sets are used to demonstrate how this aids the data-analyst in interpreting loading plots by indicating degree of significance for each variable in the plot. The usefulness of correlation loadings to visualise correlation structures between variables is also demonstrated.

The use and misuse of chemometrics for treating classification problems

In this article, we examine the increasing use by analytical chemists of chemometric methods for treating classification problems. The methods considered are principal component analysis (PCA), canonical variates analysis (CVA), discriminant analysis (DA), and discriminant partial least squares (PLS). Overfitting, a potential hazard of multivariate modelling, is illustrated using examples of real and simulated data, and the importance of model validation is discussed.

Serologic determinants of survival in patients with head and neck cancer: validating a clinical prediction model.

Quantitative measurements of serum C1q-binding macromolecules (C1qBM) and immunoglobulin A (IgA) were done on 162 patients using previously described methodology. The measurements were compared to a previously described head and neck cancer population. Using the Cox Proportional Hazards model, the prognostic implications regarding high C1qBM and subsequent death with disease (P = .02), and regional recurrence (P = .0094) were validated, but not our previous IgA-related prognostic implications. When both study populations were combined, C1qBM was predictive of survival in those patients treated with induction chemotherapy (P = .0001). C1qBM was not a significant predictor of survival in patients treated with surgery plus postoperative radiation therapy in either this second "test" population or in the original "training" population. The findings demonstrate the confounding influence of treatment modalities and the importance of model validation.

A Multichamber Model For Assessing Consumer Inhalation Exposure

The Multichamber Consumer Exposure Model (MCCEM) is a user-friendly computer program that can be used to estimate indoor air concentrations and occupant inhalation exposures for chemicals released from products, materials, furnishings or appliances in structures such as residences. Among the major features of MCCEM are flexibility in running the model for durations from one hour to one year, a library of infiltration and interzonal airflow measurements for several hundred U.S. residences, a spreadsheet for input of time-varying emission rates, the ability to estimate inhalation exposure, and options for Monte Carlo simulation and sensitivity analysis. An illustrative application is given whereby assumptions concerning the emission profile and the number of chambers to be modeled are examined for impact on exposure estimates. Discussion is provided on the importance of model validation and an approach to validation is given for the illustrative application.

Are mathematical models useful for understanding water acidification?

Computer models play an integral part in the effort to understand the complicated processes which affect ecosystems subjected to acid precipitation. Current research work using the Birkenes model attempts to improve the evaluation of the present model structure and subjects the program to a stringent verification process. Simulation results for years which are not included in, and vary considerably from, the calibration period are described. The importance of model validation is emphasized. The simulations, though in general in fair agreement with the observations, are still deficient in some respects and possible model improvements are discussed. In addition, problems which appeared during model validation are also found during verification procedures. However, computer simulations have provided the chemist with a powerful tool for evaluating current thinking regarding chemical processes involved in environmental problems such as acid precipitation.