Prediction Performance Of Model Research Articles

A prediction model for stillbirth based on first trimester pre-eclampsia combined screening.

To evaluate the accuracy of combined models of maternal biophysical factors, ultrasound, and biochemical markers for predicting stillbirths. A retrospective cohort study of pregnant women undergoing first-trimester pre-eclampsia screening at 11-13 gestational weeks was conducted. Maternal characteristics and history, mean arterial pressure (MAP) measurement, uterine artery pulsatility index (UtA-PI) ultrasound, maternal ophthalmic peak ratio Doppler, and placental growth factor (PlGF) serum were collected during the visit. Stillbirth was classified as placental dysfunction-related when it occurred with pre-eclampsia or birth weight <10th percentile. Combined prediction models were developed from significant variables in stillbirths, placental dysfunction-related, and controls. We used the area under the receiver-operating-characteristics curve (AUC), sensitivity, and specificity based on a specific cutoff to evaluate the model's predictive performance by measuring the capacity to distinguish between stillbirths and live births. There were 13 (0.79%) cases of stillbirth in 1643 women included in the analysis. The combination of maternal factors, MAP, UtA-PI, and PlGF, significantly contributed to the prediction of stillbirth. This model was a good predictor for all (including controls) types of stillbirth (AUC 0.879, 95% CI: 0.799-0.959, sensitivity of 99.3%, specificity of 38.5%), and an excellent predictor for placental dysfunction-related stillbirth (AUC 0.984, 95% CI: 0.960-1.000, sensitivity of 98.5, specificity of 85.7). Screening at 11-13 weeks' gestation by combining maternal factors, MAP, UtA-PI, and PlGF, can predict a high proportion of stillbirths. Our model has good accuracy for predicting stillbirths, predominantly placental dysfunction-related stillbirths.

Fast retrieval of XCO2 over east Asia based on Orbiting Carbon Observatory-2 (OCO-2) spectral measurements

Abstract. The increase in greenhouse gas concentrations, particularly CO2, has significant implications for global climate patterns and various aspects of human life. Spaceborne remote sensing satellites play a crucial role in high-resolution monitoring of atmospheric CO2. However, the next generation of greenhouse gas monitoring satellites is expected to face challenges, particularly in terms of computational efficiency in atmospheric CO2 retrieval and analysis. To address these challenges, this study focuses on improving the speed of retrieving the column-averaged dry-air mole fraction of carbon dioxide (XCO2) using spectral data from the Orbiting Carbon Observatory-2 (OCO-2) satellite while still maintaining retrieval accuracy. A novel approach based on neural network (NN) models is proposed to tackle the nonlinear inversion problems associated with XCO2 retrievals. The study employs a data-driven supervised learning method and explores two distinct training strategies. Firstly, training is conducted using experimental data obtained from the inversion of the operational optimization model, which is released as the OCO-2 satellite products. Secondly, training is performed using a simulated dataset generated by an accurate forward calculation model. The inversion performance and prediction performance of the machine learning model for XCO2 are compared, analyzed, and discussed for the observed region over east Asia. The results demonstrate that the model trained on simulated data accurately predicts XCO2 in the target area. Furthermore, when compared to OCO-2 satellite product data, the developed XCO2 retrieval model not only achieves rapid predictions (&lt;1 ms) with good accuracy (1.8 ppm or approximately 0.45 %) but also effectively captures sudden increases in XCO2 plumes near industrial emission sources. The accuracy of the machine learning model retrieval results is validated against reliable data from Total Carbon Column Observing Network (TCCON) sites, demonstrating its ability to effectively capture CO2 seasonal variations and annual growth trends.

Homogeneous Ensemble Feature Selection for Mass Spectrometry Data Prediction in Cancer Studies

The identification of important proteins is critical for the medical diagnosis and prognosis of common diseases. Diverse sets of computational tools have been developed for omics data reduction and protein selection. However, standard statistical models with single-feature selection involve the multi-testing burden of low power with limited available samples. Furthermore, high correlations among proteins with high redundancy and moderate effects often lead to unstable selections and cause reproducibility issues. Ensemble feature selection in machine learning (ML) may identify a stable set of disease biomarkers that could improve the prediction performance of subsequent classification models and thereby simplify their interpretability. In this study, we developed a three-stage homogeneous ensemble feature selection (HEFS) approach for both identifying proteins and improving prediction accuracy. This approach was implemented and applied to ovarian cancer proteogenomics datasets comprising (1) binary putative homologous recombination deficiency (HRD)- positive or -negative samples; (2) multiple mRNA classes (differentiated, proliferative, immunoreactive, mesenchymal, and unknown samples). We conducted and compared various ML methods with HEFS including random forest (RF), support vector machine (SVM), and neural network (NN) for predicting both binary and multiple-class outcomes. The results indicated that the prediction accuracies varied for both binary and multiple-class classifications using various ML approaches with the proposed HEFS method. RF and NN provided better prediction accuracies than simple Naive Bayes or logistic models. For binary outcomes, with a sample size of 122 and nine selected prediction proteins using our proposed three-stage HEFS approach, the best ensemble ML (Treebag) achieved 83% accuracy, 85% sensitivity, and 81% specificity. For multiple (five)-class outcomes, the proposed HEFS-selected proteins combined with Principal Component Analysis (PCA) in NN resulted in prediction accuracies for multiple-class classifications ranging from 75% to 96% for each of the five classes. Despite the different prediction accuracies of the various models, HEFS identified consistent sets of proteins linked to the binary and multiple-class outcomes.

A deep neural network prediction method for diabetes based on Kendall's correlation coefficient and attention mechanism.

Diabetes is a chronic disease, which is characterized by abnormally high blood sugar levels. It may affect various organs and tissues, and even lead to life-threatening complications. Accurate prediction of diabetes can significantly reduce its incidence. However, the current prediction methods struggle to accurately capture the essential characteristics of nonlinear data, and the black-box nature of these methods hampers its clinical application. To address these challenges, we propose KCCAM_DNN, a diabetes prediction method that integrates Kendall's correlation coefficient and an attention mechanism within a deep neural network. In the KCCAM_DNN, Kendall's correlation coefficient is initially employed for feature selection, which effectively filters out key features influencing diabetes prediction. For missing values in the data, polynomial regression is utilized for imputation, ensuring data completeness. Subsequently, we construct a deep neural network (KCCAM_DNN) based on the self-attention mechanism, which assigns greater weight to crucial features affecting diabetes and enhances the model's predictive performance. Finally, we employ the SHAP model to analyze the impact of each feature on diabetes prediction, augmenting the model's interpretability. Experimental results show that KCCAM_DNN exhibits superior performance on both PIMA Indian and LMCH diabetes datasets, achieving test accuracies of 99.090% and 99.333%, respectively, approximately 2% higher than the best existing method. These results suggest that KCCAM_DNN is proficient in diabetes prediction, providing a foundation for informed decision-making in the diagnosis and prevention of diabetes.

Predicting risk of the subsequent early pregnancy loss in women with recurrent pregnancy loss based on preconception data

BackgroundFor women who have experienced recurrent pregnancy loss (RPL), it is crucial not only to treat them but also to evaluate the risk of recurrence. The study aimed to develop a risk predictive model to predict the subsequent early pregnancy loss (EPL) in women with RPL based on preconception data.MethodsA prospective, dynamic population cohort study was carried out at the Second Hospital of Lanzhou University. From September 2019 to December 2022, a total of 1050 non-pregnant women with RPL were participated. By December 2023, 605 women had subsequent pregnancy outcomes and were randomly divided into training and validation group by 3:1 ratio. In the training group, univariable screening was performed on RPL patients with subsequent EPL outcome. The least absolute shrinkage and selection operator (LASSO) regression and multivariate logistic regression were utilized to select variables, respectively. Subsequent EPL prediction model was constructed using generalize linear model (GLM), gradient boosting machine (GBM), random forest (RF), and deep learning (DP). The variables selected by LASSO regression and multivariate logistic regression were then established and compared using the best prediction model. The AUC, calibration curve, and decision curve (DCA) were performed to assess the prediction performances of the best model. The best model was validated using the validation group. Finally, a nomogram was established based on the best predictive features.ResultsIn the training group, the GBM model achieved the best performance with the highest AUC (0.805). The AUC between the variables screened by the LASSO regression (16-variables) and logistic regression (9-variables) models showed no significant difference (AUC: 0.805 vs. 0.777, P = 0.1498). Meanwhile, the 9-variable model displayed a well discrimination performance in the validation group, with an AUC value of 0.781 (95%CI 0.702, 0.843). The DCA showed the model performed well and was feasible for making beneficial clinical decisions. Calibration curves revealed the goodness of fit between the predicted values by the model and the actual values, the Hosmer–Lemeshow test was 7.427, and P = 0.505.ConclusionsPredicting subsequent EPL in RPL patients using the GBM model has important clinical implications. Future prospective studies are needed to verify the clinical applicability.Trial registrationThis study was registered in the Chinese Clinical Trial Registry with the registration number of ChiCTR2000039414 (27/10/2020).

Computer vision for eye diseases detection using pre‐trained deep learning techniques and raspberry Pi

AbstractEarly diagnosis of eye diseases is very important to prevent visual impairment and guide appropriate treatment methods. This paper presents a unique approach that can detect numerous eye diseases automatically. Initially, this approach used the pre‐trained ImageNet models that provides various pre‐trained models for training the acquired data. The existing data sets are composed of 645 data images acquired clinically, represented by two groups of subjects as healthy and others holding the proposed eye defect like cataracts, foreign bodies, glaucoma, subconjunctival haemorrhage, and viral conjunctivitis. Followed by comparisons of the pre‐trained model's coefficients and prediction performance. Later, the first‐class execution model is integrated within the Raspberry Pi staging and the real‐time digital camera detection. The evaluation process used the confusion matrix, model accuracy, precision factor, recall coefficient, F1 score, and the Matthews Correlation Coefficient (MCC). Resulting in the performance of these pre‐trained ImageNet models used in this study represented by 93% (InceptionResNetV2), 90% (MobileNet), 86% (Residual Network ResNet50), 85% (InceptionV3), 78% (Visual Geometry Group VGG19), and 72% (Neural Architecture Search Network NASNetMobile). The results show that the InceptionResNetV2 achieved the highest performance. This proposed approach shows its efficiency and strength by early detection of the subject's unhealthy eyes through real‐time monitoring in the field of ophthalmology.

Multi-site solar irradiance prediction based on hybrid spatiotemporal graph neural network

Constructing accurate spatiotemporal correlations is a challenging task in joint prediction of multiple photovoltaic sites. Some advanced algorithms for incorporating other surrounding site information have been proposed, such as graph neural network-based methods, which are usually based on static or dynamic graphs to build spatial dependencies between sites. However, the possibility of the simultaneous existence of multiple spatial dependencies is not considered. This paper establishes a spatiotemporal prediction model based on hybrid spatiotemporal graph neural network. In this model, we apply adaptive hybrid graph learning to learn composite spatial correlations among multiple sites. A temporal convolution module with multi-subsequence temporal data input is used to extract local semantic information to better predict future nonlinear temporal dependencies. A spatiotemporal adaptive fusion module is added to address the issue of integrating diverse spatiotemporal trends among multiple sites. To assess the model's predictive performance, nine solar radiation observation stations were selected in two different climatic environments. The average root mean square error (RMSE) of the constructed model was 38.51 and 49.90 W/m2, with average mean absolute error (MAE) of 14.72 and 23.06 W/m2, respectively. Single-site and multi-site prediction models were selected as baseline models. Compared with the baseline models, the RMSE and MAE reduce by 3.1%–20.8% and 8.9%–32.8%, respectively, across all sites. The proposed model demonstrates the effectiveness of improving accuracy in forecasting solar irradiance through multi-site predictions.

Artificial neural-network-based subgrid-scale model for large-eddy simulation of isotropic turbulence

This study is concerned with accurately predicting the subgrid-scale (SGS) stress using an artificial neural network (ANN) with a linear eddy-viscosity term and a nonlinear term as the input variables. A priori and a posteriori tests are conducted to examine the prediction performance of the ANN-based SGS stress model in decaying homogeneous isotropic turbulence. In a priori test, the present ANN-based SGS model shows high correlation coefficients between the true and predicted SGS stresses, and excellent predictions of the SGS stress and dissipation. In a posteriori test, it is found that the ANN-based SGS model can predict the turbulence statistics more accurately than the traditional dynamic SGS models. The generalization capabilities of the model to untrained flow conditions and unstrained types of turbulent flow have been evaluated. It is found that the proposed ANN-based model can provide an accurate prediction of the SGS stress under different Reynolds numbers and flow types. A comparison among several existing ANN-based models with different input variables is presented, demonstrating a significant advantage of the present model.

Optimization of Power Prediction of BP Network with Improved Pelican Algorithm

Abstract In view of the large fluctuation of photovoltaic output power affected by different weather, accurate prediction of photovoltaic output power is particularly important for the safe and stable operation of power system. Firstly, the pelican optimization algorithm ( POA ) is improved in the following three aspects : adding Circle chaotic map to make the population evenly distributed, introducing mutation factor to expand the search range of pelican when approaching prey, adding adaptive weight and firefly disturbance to avoid falling into local optimum in the water surface flight stage ; then, in order to improve the prediction accuracy of BP algorithm, the improved pelican algorithm ( IPOA ) is used to optimize the weights and thresholds of BP neural network, and the IPOA-BP photovoltaic power prediction model is built to improve the accuracy of power prediction. Finally, this paper tests the prediction performance of IPOA-BP, POA-BP and basic BP power prediction models in sunny, cloudy and rainy days through experiments. The experimental results demonstrate that the IPOA-BP prediction model outperforms both the POA-BP and traditional BP neural network models under various weather conditions.

Enhancing fluid classification using meta-learning and transformer through small-sample drilling data to interpret well logging data

As geological exploration and oil and gas development continue to advance, improvement in fluid prediction becomes increasingly crucial. Drilling data often suffer from limited sample size, challenging traditional machine learning methods to fully harness these data. Consequently, a more adaptable and versatile approach is required. In response to this issue, we introduce the meta-ViT (Vision Transformer) method—a novel framework that merges meta-learning with the ViT. Meta-learning's parameter updating mechanism refines the model's ability to discern patterns and nuances across tasks, while ViT, powered by meta-learning, achieves an enhanced grasp of geological exploration characteristics, boosting fluid detection efficiency. The support set supplies meta-learning insights, while the query set assesses generalization. ViT excels at identifying subterranean fluids. Meta-learning replicates varied tasks and data distributions, fortifying model adaptability. Meanwhile, Transformers' self-attention mechanism captures distant dependencies that traditional long short-term memory struggle to manage. Their residual connections and layer normalization also address gradient challenges, simplifying training. Hence, our model effectively interprets intricate drilling data features, improving predictive accuracy and adaptability. In our experiments using a small drilling data sample set, we compared meta-ViT against other models. The results reveal superior performance of our model with limited data, affirming its efficacy and prominence in fluid classification tasks. Overall, our proposed solution excels in fluid classification tasks involving small-sample drilling data by utilizing available samples to enhance model adaptability and predictive performance.

Crowdfunding performance prediction using feature‐selection‐based machine learning models

AbstractBackgroundCrowdfunding is increasingly favoured by entrepreneurs for online financing. Predicting crowdfunding success can provide valuable guidance for stakeholders. It is a new attempt to evaluate the relative performance of different machine learning algorithms for crowdfunding prediction.ObjectivesThis study aims to identify the key factors of crowdfunding, and find the different performance and usage of machine learning algorithms for crowdfunding prediction.MethodWe crawled data from MoDian.com, a Chinese crowdfunding platform, and predicted the crowdfunding performance using four machine learning algorithms, which is a new exploration in this area. Most of the existing literature focuses on empirical analysis. This work solves the problem of predicting crowdfunding performance using a dataset with a minimal number of highly contributive features, which has higher accuracy compared to the regression analysis.ResultsThe experiment results show that feature‐selection‐based machine learning models are effective and beneficial in crowdfunding prediction.ConclusionFeature selection can significantly improve the prediction performance of the machine learning models. KNN achieved the best prediction results with five features: number of backers, target amount, number of project likes, number of project comments, and sponsor fans. The prediction accuracy was improved by 16%, the precision was improved by 13.23%, the recall was improved by 22.66%, the F‐score was improved by 18.48%, and the AUC was improved by 14.9%.

An efficient ensemble learning method based on multi-objective feature selection

Ensemble learning (EL) boosts model prediction performance across various domains through two main steps: generating individual classifiers (ICs) and combining them. Creating accurate and diverse ICs is crucial for a strong ensemble, while selecting the best ICs, known as ensemble selection (ES), is critical yet challenging due to the accuracy-diversity trade-off and the lack of agreed-upon diversity metrics. This paper introduces an EL strategy that uses multi-objective feature selection (MOFS) and a feature relevance-guided selection to tackle these challenges. Our approach uses a hybrid MOFS algorithm to produce accurate and diverse ICs, and then it employs a novel knowledge-based feature-relevance-guided metric for precise diversity assessment during ES. The ES issue is cast as an optimization problem, aiming to maximize both diversity and accuracy, and an efficient ES algorithm is developed to select optimal ICs. Extensive tests on public datasets and a real-world prediction task demonstrate the effectiveness of our method, especially in achieving high accuracy.

Drug safety assessment by machine learning models

ABSTRACT The evaluation of drug-induced Torsades de pointes (TdP) risks is crucial in drug safety assessment. In this study, we discuss machine learning approaches in the prediction of drug-induced TdP risks using preclinical data. Specifically, a random forest model was trained on the dataset generated by the rabbit ventricular wedge assay. The model prediction performance was measured on 28 drugs from the Comprehensive In Vitro Proarrhythmia Assay initiative. Leave-one-drug-out cross-validation provided an unbiased estimation of model performance. Stratified bootstrap revealed the uncertainty in the asymptotic model prediction. Our study validated the utility of machine learning approaches in predicting drug-induced TdP risks from preclinical data. Our methods can be extended to other preclinical protocols and serve as a supplementary evaluation in drug safety assessment.

Classification of autonomous vehicle crash severity: Solving the problems of imbalanced datasets and small sample size

Only a few researchers have shown how environmental factors and road features relate to Autonomous Vehicle (AV) crash severity levels, and none have focused on the data limitation problems, such as small sample sizes, imbalanced datasets, and high dimensional features. To address these problems, we analyzed an AV crash dataset (2019 to 2021) from the California Department of Motor Vehicles (CA DMV), which included 266 collision reports (51 of those causing injuries). We included external environmental variables by collecting various points of interest (POIs) and roadway features from Open Street Map (OSM) and Data San Francisco (SF). Random Over-Sampling Examples (ROSE) and the Synthetic Minority Over-Sampling Technique (SMOTE) methods were used to balance the dataset and increase the sample size. These two balancing methods were used to expand the dataset and solve the small sample size problem simultaneously. Mutual information, random forest, and XGboost were utilized to address the high dimensional feature and the selection problem caused by including a variety of types of POIs as predictive variables. Because existing studies do not use consistent procedures, we compared the effectiveness of using the feature-selection preprocessing method as the first process to employing the data-balance technique as the first process.Our results showed that AV crash severity levels are related to vehicle manufacturers, vehicle damage level, collision type, vehicle movement, the parties involved in the crash, speed limit, and some types of POIs (areas near transportation, entertainment venues, public places, schools, and medical facilities). Both resampling methods and three data preprocessing methods improved model performance, and the model that used SMOTE and data-balancing first was the best. The results suggest that over-sampling and the feature selection method can improve model prediction performance and define new factors related to AV crash severity levels.

Optimized landslide susceptibility prediction based on SBAS-InSAR: case study of the Jiuzhaigou Ms7.0 earthquake

Earthquake-induced landslides can cause severe surface damage and casualties, posing a serious threat to the overall ecological environment and social stability. Traditional landslide susceptibility prediction (LSP) techniques often suffer from low effectiveness and precision, necessitating the exploration of remote sensing technology. However, this research in this area is limited, and the development of high-performance prediction models remains a pressing scientific issue. This study focuses on the Ms7.0 earthquake in Jiuzhaigou on 8 August 2017. To investigate the optimal integration of remote sensing technology with traditional LSP techniques, the study applies collaborative factor analysis and contingency matrix methods to create four new coupling models (SVM-I, SVM-II, RF-I, RF-II), followed by a comprehensive performance evaluation of these models. The results indicate that the integration of SAR-derived surface deformation data significantly enhances the accuracy of Landslide Susceptibility Mapping (LSM). Comparing the model performance with the receiver operating characteristic curve and landslide density, the reliability and prediction performance of the RF-I model are outstanding, reflecting that the improved method based on the InSAR collaborative machine learning model with shape variables along the slope direction can optimize the accuracy of the LSM, and has better performance and robustness in earthquake landslide susceptibility evaluation.

A comparison of several cyclo-stationary and fractionally differenced time-varying autoregressive models in runoff simulation and prediction

Due to the far-reaching effects of climate change and human activities, hydrological processes exhibit cyclo-stationarity and long-term memory. Neither the traditional autoregressive (AR) nor the time-varying autoregressive (TVAR) model can comprehensively characterize these runoff characteristics, leading to poor model prediction performance. We propose a novel fractionally differenced TVAR model based on the de-seasonality method (FDTVARD) to better capture the above runoff characteristics and achieve improved runoff simulation and prediction accuracy. We first compare the effects of two covariates, time and lagged target variables, on the performance of the FDTVARD model. The FDTVARD model is compared with the AR model based on the de-seasonality method (ARD), fractionally differenced ARD (FDARD) and TVAR model based on the de-seasonality method (TVARD), and the model performance is evaluated using daily runoff data from seven hydrological stations in the Yellow River Basin, China. The results show that (1) the FDTVARD model has the simplest model form and yields the best simulation and forecasting performance, and the optimal covariate is time. (2) Considering only the time-varying nature of the model coefficients improves the ARD model prediction performance more than considering only the long-term memory of the runoff series. Considering both factors improve the simulation and prediction performance of the ARD model more than considering only one factor. The study findings could facilitate improvements in the prediction accuracy of stochastic hydrological models.

The advanced lung cancer inflammation index (ALI) predicted the postoperative survival rate of patients with non-small cell lung cancer and the construction of a nomogram model

ObjectiveTo investigate the prognostic significance of the advanced lung cancer inflammation index (ALI) in patients with operable non-small-cell lung carcinoma (NSCLC). By constructing the nomogram model, it can provide a reference for clinical work.MethodsA total of 899 patients with non-small cell lung cancer who underwent surgery in our hospital between January 2017 and June 2021 were retrospectively included. ALI was calculated by body mass index (BMI) × serum albumin/neutrophil to lymphocyte ratio (NLR). The optimal truncation value of ALI was obtained using the receiver operating characteristic (ROC) curve and divided into two groups. Survival analysis was represented by the Kaplan-Meier curve. The predictors of Overall survival (OS) were evaluated by the Cox proportional risk model using single factor and stepwise regression multifactor analysis. Based on the results of multi-factor Cox proportional risk regression analysis, a nomogram model was established using the R survival package. The bootstrap method (repeated sampling 1 000 times) was used for internal verification of the nomogram model. The concordance index (C-index) was used to represent the prediction performance of the nomogram model, and the calibration graph method was used to visually represent its prediction conformity. The application value of the model was evaluated by decision curve analysis (DCA).ResultsThe optimal cut-off value of ALI was 70.06, and the low ALI group (ALI < 70.06) showed a poor survival prognosis. In multivariate analyses, tumor location, pathological stage, neuroaggression, and ALI were independently associated with operable NSCLC-specific survival. The C index of OS predicted by the nomogram model was 0.928 (95% CI: 0.904–0.952). The bootstrap self-sampling method (B = 1000) was used for internal validation of the prediction model, and the calibration curve showed good agreement between the prediction and observation results of 1-year, 2-year, and 3-year OS. The ROC curves for 1-year, 2-year, and 3-year survival were plotted according to independent factors, and the AUC was 0.952 (95% CI: 0.925–0.979), 0.951 (95% CI: 0.916–0.985), and 0.939 (95% CI: 0.913–0.965), respectively. DCA shows that this model has good clinical application value.ConclusionALI can be used as a reliable indicator to evaluate the prognosis of patients with operable NSCLC, and through the construction of a nomogram model, it can facilitate better individualized treatment and prognosis assessment.

Automatic cross-validation in structured models: Is it time to leave out leave-one-out?

Standard techniques such as leave-one-out cross-validation (LOOCV) might not be suitable for evaluating the predictive performance of models incorporating structured random effects. In such cases, the correlation between the training and test sets could have a notable impact on the model’s prediction error. To overcome this issue, an automatic group construction procedure for leave-group-out cross validation (LGOCV) has recently emerged as a valuable tool for enhancing predictive performance measurement in structured models. The purpose of this paper is (i) to compare LOOCV and LGOCV within structured models, emphasizing model selection and predictive performance, and (ii) to provide real data applications in spatial statistics using complex structured models fitted with INLA, showcasing the utility of the automatic LGOCV method. First, we briefly review the key aspects of the recently proposed LGOCV method for automatic group construction in latent Gaussian models. We also demonstrate the effectiveness of this method for selecting the model with the highest predictive performance by simulating extrapolation tasks in both temporal and spatial data analyses. Finally, we provide insights into the effectiveness of the LGOCV method in modeling complex structured data, encompassing spatio-temporal multivariate count data, spatial compositional data, and spatio-temporal geospatial data.

Evaluating the sensitivity and robustness of occupancy models for building energy simulation during design

Occupancy modeling plays a crucial role in building energy simulation in building design, yet studies investigating the influence of model types and their input assumptions remain sparse. This study evaluates the sensitivity of parameters and model robustness to uncertainties in parameter ranges and data availability during the design phase. We assessed three occupancy models—deterministic, probabilistic, and agent-based—using an open dataset with occupancy data across four space types. Our findings reveal that the predictive performance of deterministic and probabilistic models is influenced heavily by a few of the most sensitive parameters. In contrast, more parameters affected the prediction performance of the agent-based model. This suggests a need to prioritize sensitive parameter estimation to enhance prediction accuracy in deterministic and probabilistic models. Specifically, the occupancy density and start/end time, followed by schedule fractions, emerged as highly sensitive due to their significant impact on average occupancy profiles. We also found that for hourly energy predictions, narrowing the range of the top five sensitive parameters decreases the coefficient of variation of the root mean square error (CV(RMSE)) by 3% and reduces its standard deviation by half. These results underscore the importance of careful parameter estimation to improve the reliability of occupancy models in building energy simulations.

Evaluating the reliability of machine-learning-based predictions used in nuclear power plant instrumentation and control systems

The field of data-driven, neural-network-based machine learning (ML) has seen significant growth, with applications in various information and control systems. Despite promising real-world uses, the reliability of models remains questionable. Conventionally, reliability is assessed based on predictive fidelity, accuracy, and training effectiveness; however, quality developmental procedures and excellent training performance metrics do not guarantee operational reliability. Instead, an ML model's predictive performance depends on the training set's representativeness to the intended operational space. It is known that ML algorithms excel at interpolation but struggle with extrapolation tasks. Anomalies and feature drift can also reduce operational performance. Determining whether a new sample is an interpolation or extrapolation task involves out-of-distribution (OOD) detection for assessing its proximity to the existing training data. Thus, we present a real-time, model-agnostic individual prediction reliability evaluation method called Data Auditing for Reliability Evaluation (DARE) for applying OOD detection to the training dataset. We demonstrate on a feedforward neural network ML-integrated digital twin for predicting fuel centerline temperatures during loss-of-flow transients. DARE acts as a “data supervisor” in determining the model's applicability under different operating conditions. In this manner, we demonstrate how training data can serve as inductive evidence to support the reliability of ML predictions.