Perform Model Validation Research Articles

A Novel Machine Learning Model and a Web Portal for Predicting the Human Skin Sensitization Effects of Chemical Agents

Skin sensitization is a significant concern for chemical safety assessments. Traditional animal assays often fail to predict human responses accurately, and ethical constraints limit the collection of human data, necessitating a need for reliable in silico models of skin sensitization prediction. This study introduces HuSSPred, an in silico tool based on the Human Predictive Patch Test (HPPT). HuSSPred aims to enhance the reliability of predicting human skin sensitization effects for chemical agents to support their regulatory assessment. We have curated an extensive HPPT database and performed chemical space analysis and grouping. Binary and multiclass QSAR models were developed with Bayesian hyperparameter optimization. Model performance was evaluated via five-fold cross-validation. We performed model validation with reference data from the Defined Approaches for Skin Sensitization (DASS) app. HuSSPred models demonstrated strong predictive performance with CCR ranging from 55 to 88%, sensitivity between 48 and 89%, and specificity between 37 and 92%. The positive predictive value (PPV) ranged from 84 to 97%, versus negative predictive value (NPV) from 22 to 65%, and coverage was between 75 and 93%. Our models exhibited comparable or improved performance compared to existing tools, and the external validation showed the high accuracy and sensitivity of the developed models. HuSSPred provides a reliable, open-access, and ethical alternative to traditional testing for skin sensitization. Its high accuracy and reasonable coverage make it a valuable resource for regulatory assessments, aligning with the 3Rs principles. The publicly accessible HuSSPred web tool offers a user-friendly interface for predicting skin sensitization based on chemical structure.

Validation of predictive performance models for supersonic gas-jet nozzles at the Laboratory for Laser Energetics.

We present results characterizing the neutral-density distributions produced by the supersonic nozzles used in experiments on the OMEGA-60 and OMEGA-EP laser systems at the University of Rochester's Laboratory for Laser Energetics (LLE). Axisymmetric Fluent® simulations using LLE nozzle specifications capture the viscous effects, gas expansion, and shock waves that complicate flow predictions for offsets above the nozzle exit. These simulations show good agreement with neutral-density measurements obtained using a four-wave shearing interferometer. An analytical form is given for the plateau length. Fits to simulation data for boundary layer thickness, mean plateau density, and density ramps are given as functions of nozzle offset and nozzle backing pressure for a number of nozzles and gases.

An Application of Artificial Neural Network for Predicting Threshing Performance in a Flexible Threshing Device

Rice is a widely cultivated food crop worldwide, and threshing is one of the most important operations of combine harvesters in grain production. It is a complex, nonlinear, multi-parameter physical process. The flexible threshing device has unique advantages in reducing the grain damage rate and has already been one of the major concerns in engineering design. Using the measured test database of the flexible threshing test bench, the rotation speed of the threshing cylinder (RS), threshing clearance of the concave sieve (TC), separation clearance of the concave sieve (SC), and feeding quantity (FQ) are used as the input layer. In contrast, the crushing rate (YP), impurity rate of the threshed material (YZ), and loss rate (YS) are used in the output layer. A 4-5-3-3 artificial neural network (ANN) model, with a backpropagation learning algorithm, was developed to predict the threshing performance of the flexible threshing device. Next, we explored the degree to which the inputs affect the outputs. The results showed that the R of the threshing performance model validation set in the hidden layer reached 0.980, and the root mean square error (RMSE) and the average absolute error (MAE) were less than 0.139 and 0.153, respectively. The built neural network model predicted the performance of the flexible threshing device, and the regression determination coefficient R2 between the prediction data and the experimental data was 0.953. The results showed revealed that the data combined with the ANN method is an effective approach for predicting the threshing performance of the flexible threshing device in rice. Moreover, the sensitivity analysis showed that RS, TC, and SC were crucial factors influencing the performance of the flexible threshing device, with an average relative importance of 15.00%, 14.89%, and 14.32%, respectively. FQ had the least effect on threshing performance, with an average threshing relative importance of 11.65%. Our findings can be leveraged to optimize the threshing performance of future flexible threshing devices.

Characterizing spatial variability in the temperature field to support thermal model validation in a naturally ventilated building

ABSTRACT Night-time passive cooling is an energy-efficient cooling strategy, but the design of passive cooling systems relies on imperfect computational models, which require validation. This paper assesses the importance of spatial variability in the temperature field when performing model validation. Full-scale temperature measurements in a three story atrium building reveal spatial variability of up to on each floor during the natural ventilation process. Validation of a dynamic thermal model with uncertainty quantification reveals accurate volume-averaged air temperature predictions. Discrepancies are on the order of the sensor accuracy (), and are primarily due to slightly under-predicted cooling rates in the model. Importantly, this trend would be identified incorrectly when validating the model against the building's built-in sensors, which consistently record 0.05–1.63 higher temperatures than the volume-averaged air temperature. These findings highlight the importance of spatial variability and careful temperature sensor placement in naturally ventilated buildings.

Drought and water availability analysis for irrigation and household water needs in the Krueng Jrue sub-watershed.

This study aimed to analyze drought conditions and evaluate irrigation water availability and household water needs in the Krueng Jrue sub-watershed, Aceh Province, Indonesia. The Z-score statistics method was developed to analyze the drought, and the Mock model was used to generate discharges. We performed model validation using linear regression, which produced a coefficient of determination (R 2 = 0.90**) and coefficient of regression (r = 0.95**). In general, this area had a normal Z-score for precipitation (ZSP) class with 90 events (75%) and a normal Z-score for a discharge (ZSD) class with 89 events (74.2%). There were 0-11 (0-9.2%) moderate wet, very wet, extreme wet, moderate drought, and severe drought events. The consistency between the ZSP and ZSD indices reached 85.8%, indicating consensus between the meteorological droughts that were analyzed based on rainfall (ZSP) and hydrological droughts analyzed based on water discharge (ZSD). ZSP and ZSD indices showed negative values during the dry season (April to September) and positive values during the rainy season (October to March). There was a surplus of water availability for irrigation and household water needs during the rainy season and a deficit during the dry season. However, water deficits also occurred in certain months during the rainy rendeng planting season, for example, in October 2009, 2013, 2016, and 2017 as well as in February between 2008 to 2011 and from 2014 to 2017. This observation was probably due to the influence of global climate variables that need to be substantiated. This study offers necessary information for farmers, the community, and the local government when anticipating drought phenomenon, organizing the rice planting season, and evaluating water availability in other watersheds.

PROBE VEHICLE LANE IDENTIFICATION AT SIGNALIZED INTERSECTION USING SUPPORT VECTOR MACHINE METHOD

Transportation problems in Indonesia, especially in urban areas, can be solved by applying Intelligent Transportation Systems (ITS), one of which is by using a Global Positioning System (GPS) based probe vehicle system. This system is growing because smartphones can be used as instruments for investigators' vehicles. This system provides information on the location and speed of all investigator vehicles every second, making it possible to estimate the real time traffic condition on a road segment passed by the probe vehicle. The implementation of this system in the city of Manado can help improve traffic conditions because so many motorists carry smartphones while driving. However, the problem is that the GPS on this smartphone has a low accuracy of 3 to 15 meters, so this GPS data cannot be used to solve transportation problems at the micro level such as at intersections. The purpose of this study was to obtain a method for identifying probe vehicle lanes through GPS data from smartphones. The steps in achieving the research goal are 1) building a microscopic transportation model using software to obtain data 2) building a data processing algorithm for vehicle lane identification using traffic flow theory and machine learning methods which in this case is Support Vector Machine, and 3) perform model validation. The results showed that the individual queue shock wave from each probe vehicle could be used to identify the lane of the probe vehicle. This study also confirms that the Support Vector Machine method can be used to predict the lane position of the probe vehicle.

BetaBayes—A Bayesian Approach for Comparing Ecological Communities

Ecological communities change because of both natural and human factors. Distinguishing between the two is critical to ecology and conservation science. One of the most common approaches for modelling species composition changes is calculating beta diversity indices and then relating index changes to environmental changes. The main difficulty with these analyses is that beta diversity indices are paired comparisons, which means indices calculated with the same community are not independent. Mantel tests and generalised dissimilarity modelling (GDM) are two of the most commonly used statistical procedures for analysing such data, employing randomisation tests to consider the data’s dependence. Here, we introduce a Bayesian model-based approach called BetaBayes that explicitly incorporates the data dependence. This approach is based on the Bradley–Terry model, which is a widely used approach for modelling paired comparisons that involves building a standard regression model containing two varying intercepts, one for each community involved in the beta diversity index, that capture their respective contributions. We used BetaBayes to analyse a famous dataset collected in Panama that contains information on multiple 1 ha plots from the rain forests of Panama. We calculated the Bray–Curtis index between all pairs of plots, analysed the relationship between the index and two covariates (geographic distance and elevation), and compared the results of BetaBayes with those from the Mantel test and GDM. BetaBayes has two distinctive features. The first is its flexibility, which allows the user to quickly change it to fit the data structure; namely, by adding varying effects, incorporating spatial autocorrelation, and modelling complex nonlinear relationships. The second is that it provides a clear path for performing model validation and model improvement. BetaBayes avoids hypothesis testing, instead focusing on recreating the data generating process and quantifying all the model configurations that are consistent with the observed data.

On the Optical Characterization of Architectural Three-Dimensional Skins and Their Solar Control Potential

The use of second building skins is becoming a trademark in modern architecture, opening for innovative solutions, such as three-dimensional (3D) systems. This paper explores the potential of these systems to provide adequate solar protection to glazed façades by means of an advanced optical characterization. Spectral transmittance and reflectance of fourteen samples, belonging to several technological families, are measured with a built-in spectrophotometer, suitable to accurately characterize complex semi-transparent systems. Solar and lighting properties are then calculated. The normal optical properties strongly depend on the openness factor, thus the geometry primarily affects the performance. A total of 11 samples exhibit normal solar transmittance in the 40–53% range; the value decreases to 20% for the plissé metal grid and increases to 70% on average for metal meshes. The angular transmittance depends on the system texture geometry and its self-shading capabilities. It was found that such systems underperform as static conventional shading systems; however, one of the metal meshes, the plissé grid and the plastic grid exhibit relevant angular selectivity, with transmittance decay at 60° in the 58–72% range compared to the normal incidence value. The results show that some of the selected 3D systems provide adequate solar protection. The developed dataset can be used for early-stage design analyses, as well as for energy performance model input and validation.

Wheel wear performance assessment and model validation using Harold full scale test rig

The railway industry has focused on the improvement of maintenance through the use of novel technologies. Recently, the utilisation of repair welding to restore the worn area on wheels has been investigated, as it can bring significant savings in wheelset maintenance. Under an Innovate UK AURORA project, a worn wheel that previously operated on London Underground (LUL) was restored using this process. To test its performance and compare with a new standard steel grade wheel, the HAROLD full scale test rig was used in which an LUL vehicle bogie equipped with both the restored and R9 grade wheels was operated under the representative lateral and yaw displacements computed from vehicle dynamics simulations. The wear measurements carried out at the end of test cycles showed that although the restored wheel suffered from initial higher wear, the levels reduced and became similar to the R9 grade wheel. Furthermore, the full-scale testing provided an opportunity to validate the wear model predictions which were conducted using the vehicle dynamics simulations utilised in testing inputs. It was found that while the flange wear predictions were higher, the tread wear estimations were smaller than the measurements on the R9 grade wheel.

An investigation of fire evacuation performance in irregular underground commercial building affected by multiple parameters

Irregular underground commercial buildings (IUCB) are difficult to evacuate because of large space depth, complex form changes, and dense crowd. Fire evacuation performance of IUCB is of great practical importance with the development of underground buildings. To analyze the influence of different evacuation parameters on the properties of IUCB fire evacuation, this research gives person evacuation time under different strategies, action time of agent, and evacuation time based on IUCB fire evacuation numerical analysis model. A virtual reality system (VRS) including testing equipment and record platform is designed to perform model validation. Influence of three parameters are also analyzed on the performance of IUCB fire evacuation, including evacuation numbers, minimal group effect and pre-action time. Social force model and Generalized random graph model are applied to behaviors of evacuee. The results show that crowd congestion becomes a key factor determining evacuation time after the number of evacuees’ increases by 50.0%. The unbalanced use of exits reduces the overall evacuation rate. The average evacuation time, the overall evacuation time, and the total evacuation time of 95.0% population increase by 13.3%, 36.0%, and 15.9% respectively. When 50.0% of the evacuees are set as the minimal group effect, the overall evacuation time increased by the minimal group effect aggravates the congestion in some routes, but the minimal group effect could reduce the long tail time. Reducing the pre-action time can improve the overall evacuation rate in the early stage of evacuation. The basic linear relationship between pre-action time and evacuation time is presented. This research can provide the guidance for exit fire evacuation for IUCB to be built, which can also reduce the evacuation time in fire evacuation.

Developing prediction models for total knee replacement surgery in patients with osteoarthritis: Statistical analysis plan

Approximately 12-20% of those with osteoarthritis (OA) in Australia who undergo total knee replacement (TKR) surgery do not report any clinical improvement. There is a need to develop prediction tools for use in general practice that allow early identification of patients likely to undergo TKR and those unlikely to benefit from the surgery. First-line treatment strategies can then be implemented and optimised to delay or prevent the need for TKR. The identification of potential non-responders to TKR may provide the opportunity for new treatment strategies to be developed and help ensure surgery is reserved for those most likely to benefit. This statistical analysis plan (SAP) details the statistical methodology used to develop such prediction tools. To describe in detail the statistical methods used to develop and validate prediction models for TKR surgery in Australian patients with OA for use in general practice. This SAP contains a brief justification for the need for prediction models for TKR surgery in general practice. A description of the data sources that will be linked and used to develop the models, and estimated sample sizes is provided. The planned methodologies for candidate predictor selection, model development, measuring model performance and internal model validation are described in detail. Intended table layouts for presentation of model results are provided. Consistent with best practice guidelines, the statistical methodologies outlined in this SAP have been pre-specified prior to data pre-processing and model development.

Mapping forest age using National Forest Inventory, airborne laser scanning, and Sentinel-2 data

BackgroundThe age of forest stands is critical information for forest management and conservation, for example for growth modelling, timing of management activities and harvesting, or decisions about protection areas. However, area-wide information about forest stand age often does not exist. In this study, we developed regression models for large-scale area-wide prediction of age in Norwegian forests. For model development we used more than 4800 plots of the Norwegian National Forest Inventory (NFI) distributed over Norway between latitudes 58° and 65° N in an 18.2 Mha study area. Predictor variables were based on airborne laser scanning (ALS), Sentinel-2, and existing public map data. We performed model validation on an independent data set consisting of 63 spruce stands with known age.ResultsThe best modelling strategy was to fit independent linear regression models to each observed site index (SI) level and using a SI prediction map in the application of the models. The most important predictor variable was an upper percentile of the ALS heights, and root mean squared errors (RMSEs) ranged between 3 and 31 years (6% to 26%) for SI-specific models, and 21 years (25%) on average. Mean deviance (MD) ranged between − 1 and 3 years. The models improved with increasing SI and the RMSEs were largest for low SI stands older than 100 years. Using a mapped SI, which is required for practical applications, RMSE and MD on plot level ranged from 19 to 56 years (29% to 53%), and 5 to 37 years (5% to 31%), respectively. For the validation stands, the RMSE and MD were 12 (22%) and 2 years (3%), respectively.ConclusionsTree height estimated from airborne laser scanning and predicted site index were the most important variables in the models describing age. Overall, we obtained good results, especially for stands with high SI. The models could be considered for practical applications, although we see considerable potential for improvements if better SI maps were available.

Comparison of Model Predictions and Performance Test Data for a Prototype Thermal Energy Storage Module

Abstract Although model predictions of thermal energy storage (TES) performance have been explored in previous investigations, relevant test data that enable experimental validation of performance models have been limited. This is particularly true for high-performance TES designs that facilitate fast input and extraction of energy. In this paper, we present a summary of experimental tests of a high-performance TES unit using lithium nitrate trihydrate phase change material as a storage medium. Performance data are presented for complete dual-mode cycles consisting of extraction (melting) followed by charging (freezing). These tests simulate the cyclic operation of a TES unit for asynchronous cooling in a variety of applications. The model analysis is found to agree reasonably well, within 10%, with the experimental data except for conditions very near the initiation of freezing, a consequence of subcooling that is required to initiate solidification.

Handling missing predictor values when validating and applying a prediction model to new patients.

Missing data present challenges for development and real‐world application of clinical prediction models. While these challenges have received considerable attention in the development setting, there is only sparse research on the handling of missing data in applied settings. The main unique feature of handling missing data in these settings is that missing data methods have to be performed for a single new individual, precluding direct application of mainstay methods used during model development. Correspondingly, we propose that it is desirable to perform model validation using missing data methods that transfer to practice in single new patients. This article compares existing and new methods to account for missing data for a new individual in the context of prediction. These methods are based on (i) submodels based on observed data only, (ii) marginalization over the missing variables, or (iii) imputation based on fully conditional specification (also known as chained equations). They were compared in an internal validation setting to highlight the use of missing data methods that transfer to practice while validating a model. As a reference, they were compared to the use of multiple imputation by chained equations in a set of test patients, because this has been used in validation studies in the past. The methods were evaluated in a simulation study where performance was measured by means of optimism corrected C‐statistic and mean squared prediction error. Furthermore, they were applied in data from a large Dutch cohort of prophylactic implantable cardioverter defibrillator patients.

Aircraft Performance Model Calibration and Validation for General Aviation Safety Analysis

Performance models facilitate a wide range of safety analyses in aviation. In an ideal scenario, the performance models would show inherently good agreement with the true performance of the aircraft. However, in reality, this is rarely the case: either owing to underlying simplifications or due to the limited fidelity of applicable tools or data. In such cases, calibration is required to fine-tune the behavior of the performance models. For point-mass steady-state performance models, challenges arise due to the fact that there is no obvious, unique metric or flight condition at which to assess the accuracy of the model predictions, as well as because a large number of model parameters may potentially influence model accuracy. This work presents a two-level approach to aircraft performance model calibration. The first level consists of using manufacturer-developed performance manuals for calibration, whereas the second level provides additional refinement when flight data are available. The performance models considered in this work consist of aerodynamic and propulsion models (performance curves) that are capable of predicting the nondimensional lift, drag, thrust, and torque at any given point in time. The framework is demonstrated on two representative general aviation aircraft. The demonstrated approach results in models that can predict critical energy-based safety metrics with improved accuracy for use in retrospective safety analyses.

Sumrep: A Summary Statistic Framework for Immune Receptor Repertoire Comparison and Model Validation.

The adaptive immune system generates an incredible diversity of antigen receptors for B and T cells to keep dangerous pathogens at bay. The DNA sequences coding for these receptors arise by a complex recombination process followed by a series of productivity-based filters, as well as affinity maturation for B cells, giving considerable diversity to the circulating pool of receptor sequences. Although these datasets hold considerable promise for medical and public health applications, the complex structure of the resulting adaptive immune receptor repertoire sequencing (AIRR-seq) datasets makes analysis difficult. In this paper we introduce sumrep, an R package that efficiently performs a wide variety of repertoire summaries and comparisons, and show how sumrep can be used to perform model validation. We find that summaries vary in their ability to differentiate between datasets, although many are able to distinguish between covariates such as donor, timepoint, and cell type for BCR and TCR repertoires. We show that deletion and insertion lengths resulting from V(D)J recombination tend to be more discriminative characterizations of a repertoire than summaries that describe the amino acid composition of the CDR3 region. We also find that state-of-the-art generative models excel at recapitulating gene usage and recombination statistics in a given experimental repertoire, but struggle to capture many physiochemical properties of real repertoires.

Long-term high frequency monitoring of a large borehole heat exchanger array

Borehole heat exchangers are a key technological element of geothermal energy systems and modelling their behaviour has received much attention. The aim in the work reported here has been to produce a reference data set that can be used in analysis of large borehole heat exchanger systems and validation of models of such. A monitoring exercise to collect high frequency data from a large ground heat exchanger array consisting of 56 boreholes over 38 months since the start of operations is reported. The system is associated with a mixed-use university building that has both heating and cooling loads. Ground heat exchange was found to be dominated by rejection of heat over the monitoring period and modest seasonal increases in temperatures. The ground heat exchanger installation has been additionally characterised by analysis of thermal response test data to estimate the effective ground and grout thermal properties. The utility of the measurements as a reference data set by presenting a model validation study is furthermore demonstrated. This has highlighted some features of the data that are more significant in systems of larger scale. These reference data are being made openly available for further work on performance analysis and model validation.

Optimization and control of the light environment for greenhouse crop production

Optimization and control of the greenhouse light environment is key to increasing crop yield and quality. However, the light saturation point impacts the efficient use of light. Therefore, the dynamic acquisition of the light saturation point that is influenced by changes in temperature and CO2 concentration is an important challenge for the development of greenhouse light environment control system. In view of this challenge, this paper describes a light environment optimization and control model based on a crop growth model for predicting cucumber photosynthesis. The photosynthetic rate values for different photosynthetic photon flux densities (PPFD), CO2 concentration, and temperature conditions provided to cucumber seedlings were obtained by using an LI-6400XT portable photosynthesis system during multi-factorial experiments. Based on the measured data, photosynthetic rate predictions were determined. Next, a support vector machine(SVM) photosynthetic rate prediction model was used to obtain the light response curve under other temperatures and CO2 conditions. The light saturation point was used to establish the light environment optimization and control model and to perform model validation. The slope of the fitting straight line comparing the measured and predicted light saturation point was 0.99, the intercept was 23.46 and the coefficient of determination was 0.98. The light control model was able to perform dynamic acquisition of the light saturation point and provide a theoretical basis for the efficient and accurate control of the greenhouse light environment.

A routine blood test-associated predictive model and application for tuberculosis diagnosis: a retrospective cohort study from northwest China.

ObjectivesThis study aimed to use the results of routine blood tests and relevant parameters to construct models for the prediction of active tuberculosis (ATB) and drug-resistant tuberculosis (DRTB) and to assess the diagnostic values of these models.MethodsWe performed logistic regression analysis to generate models of plateletcrit-albumin scoring (PAS) and platelet distribution width-treatment-sputum scoring (PTS). Area under the curve (AUC) analysis was used to analyze the diagnostic values of these curves. Finally, we performed model validation and application assessment.ResultsIn the training cohort, for the PAS model, the AUC for diagnosing ATB was 0.902, sensitivity was 82.75%, specificity was 82.20%, accuracy rate was 81.00%, and optimal threshold value was 0.199. For the PTS model, the AUC for diagnosing DRTB was 0.700, sensitivity was 63.64%, specificity was 73.53%, accuracy rate was 89.00%, and optimal threshold value was −2.202. These two models showed significant differences in the AUC analysis, compared with single-factor models. Results in the validation cohort were similar.ConclusionsThe PAS model had high sensitivity and specificity for the diagnosis of ATB, and the PTS model had strong predictive potential for the diagnosis of DRTB.

A Novel Energy Harvester for Powering Small UAVs: Performance Analysis, Model Validation and Flight Results.

The proposed work aims at exploring and developing new strategies to extend mission parameters (measured as travel distance and mission duration (MD)) of a new class of unmanned vehicles, named Micro Air Vehicles (MAVs). In this paper, a new analytical model, identifying all factors, which determine the MAV power consumption, is presented. Starting from the new model, the design of a nanoarray energy harvester, based on plasmonics nano-antenna technology is proposed. The preliminary study was based on a 22,066,058 22,066,058 × 62,800-dipole rectenna array producing an output power level of 84.14 mW, and an energy value of 2572 J under a power density of 100 mW/cm² and a resonant frequency of 350 THz as input conditions. The preliminary analytical results show a possible recharge of an ultra-fast rechargeable battery on board of a MAV and an MD improvement of 16.30 min.