Model Selection Methodology Research Articles

Model benchmarking for PEM Water Electrolyzer for energy management purposes

This research conducts a comprehensive evaluation of various Proton Exchange Membrane Water Electrolyzer (PEMWE) models through a test bench, optimizing parameters and comparing obtained models against real-world data. Key operational factors such as reversible potential, activation overpotential, ohmic overpotential, and concentration overpotential are examined through experimental data. This study addresses critical gaps in current PEMWE research by reviewing modelling approaches, introducing a novel classification of models, and proposing an integrated approach that combines experimental validation with comprehensive model analysis. A novel, systematic methodology for model and submodel selection is presented, enabling practitioners to identify models that balance computational efficiency and predictive accuracy tailored to specific energy management and power allocation needs. This approach bridges the gap between complex modelling and industrial applications, enhancing the practical implementation of PEMWE systems in sustainable hydrogen production. Enhances model reliability for operational and manufacturing differences, provides invaluable guidance for improving the design and operation of these systems, and promotes a more robust and efficient hydrogen energy infrastructure.

Performance evaluation of CMIP6 models for application to hydrological modelling studies – A case study of Australia

Accurate estimation of climate change impacts on catchment hydrology is essential for effective future water management. The efficacy of such estimations is dependent on proper climate model selection. In this study, an attempt was made to formulate a methodology for climate model selection, evaluating eight climate models from the sixth phase of the Coupled Model Intercomparison Project (CMIP6). The models were assessed for their ability to simulate variables used in hydrological studies and large-scale atmospheric circulation influencing rainfall in Australia. Five statistical indicators Root Mean Square Error (RMSE), Spatial Correlation (SC), Percentage Bias (Pbias), Normalized Root Mean Square Error (NRMSE), and Nash-Sutcliffe Efficiency (NSE) were used to evaluate the performance, and the models were ranked through Compromise Programming (CP), a multiple criteria decision making technique. Results show that HadGEM3-GC31-LL performed well in most of the categories considered and was top top-ranked model overall followed by GFDL-ESM4, CESM2-CAM6-RT, and CanESM5 for Australia. Conversely, MIROC6 consistently ranked lower in most of the categories. In the context of simulating hydrological variables, CESM2-CAM6-RT, HadGEM3-GC31-LL, and GFDL-ESM4 emerged as the top three models. The robustness of the proposed methodology suggests its applicability for model selection, making it a replicable approach for climate change impact assessment studies in diverse regions.

Expected credit loss modeling

This article proposes a method for modeling the probability of default, describes the statistical evaluation of the model, and presents a model of the software implementation algorithm. The algorithm automatically selects from the group of regression models where the models are both linear regression and various modifications of semi-logarithmic models and lag models for macro factors Xi,t,Xi,t-1, ...,Xi,t-TStatistical analysis is carried out using the coefficient of determination R-squared, p-value, VIF (variance inflation factor).The relevance of this topic is determined by the need for banking organizations to comply with international standards, such as International Financial Reporting Standards (IFRS 9) and the Agreement on Banking Supervision and Capital (Basel 3). These standards define credit risk assessment requirements and capital requirements. Adherence to these standards is important not only for ensuring the stability and reliability of the financial system, but also for maintaining the trust of clients and investors. Compliance with international standards also makes banks competitive in the global market and promotes investment inflows and the development of the financial sector.IFRS 9 can be presented in various mathematical models. The article proposes an approach to choosing the appropriate model for forecasting the probability of default. The described model selection method allows banks to choose the optimal default forecast assessment model within the framework of the given standard. This contributes to a more accurate and reliable assessment of credit risk, in accordance with regulatory requirements, which will provide banks with the means for better forecasting and management of financial resources, as well as risk reduction.The model selection methodology saves a significant amount of time and resources, since the search for the optimal model occurs automatically. This allows us to react more quickly to changes in the economic environment, improve decision-making strategies and manage credit risks, which is of great importance for financial institutions in a competitive environment.There is currently a war going on in Ukraine, and forecasting using current methods becomes a difficult task due to unpredictable stressful situations for the economy. In such conditions, standard models may not be sufficiently adapted to account for increased risk and volatility. The proposed approach allows finding more conservative forecasting models that can be useful in unstable periods and war.

Improving Earth System Model Selection Methodologies for Projecting Hydroclimatic Change: Case Study in the Pacific Northwest

AbstractThe rapid expansion of Earth system model (ESM) data available from the Coupled Model Intercomparison Project Phase 6 (CMIP6) necessitates new methods to evaluate the performance and suitability of ESMs used for hydroclimate applications as these extremely large data volumes complicate stakeholder efforts to use new ESM outputs in updated climate vulnerability and impact assessments. We develop an analysis framework to inform ESM sub‐selection based on process‐oriented considerations and demonstrate its performance for a regional application in the US Pacific Northwest. First, a suite of global and regional metrics is calculated, using multiple historical observation datasets to assess ESM performance. These metrics are then used to rank CMIP6 models, and a culled ensemble of models is selected using a trend‐related diagnostics approach. This culling strategy does not dramatically change climate scenario trend projections in this region, despite retaining only 20% of the CMIP6 ESMs in the final model ensemble. The reliability of the culled trend projection envelope and model response similarity is also assessed using a perfect model framework. The absolute difference in temperature trend projections is reduced relative to the full ensemble compared to the model for each SSP scenario, while precipitation trend errors are largely unaffected. In addition, we find that the spread of the culled ensemble temperature and precipitation trends includes the trend of the “truth” model ∼83%‐92% of the time. This analysis demonstrates a reliable method to reduce ESM ensemble size that can ease use of ESMs for creating and understanding climate vulnerability and impact assessments.

Statistical framework to determine indel-length distribution.

Insertions and deletions (indels) of short DNA segments, along with substitutions, are the most frequent molecular evolutionary events. Indels were shown to affect numerous macro-evolutionary processes. Because indels may span multiple positions, their impact is a product of both their rate and their length distribution. An accurate inference of indel-length distribution is important for multiple evolutionary and bioinformatics applications, most notably for alignment software. Previous studies counted the number of continuous gap characters in alignments to determine the best-fitting length distribution. However, gap-counting methods are not statistically rigorous, as gap blocks are not synonymous with indels. Furthermore, such methods rely on alignments that regularly contain errors and are biased due to the assumption of alignment methods that indels lengths follow a geometric distribution. We aimed to determine which indel-length distribution best characterizes alignments using statistical rigorous methodologies. To this end, we reduced the alignment bias using a machine-learning algorithm and applied an Approximate Bayesian Computation methodology for model selection. Moreover, we developed a novel method to test if current indel models provide an adequate representation of the evolutionary process. We found that the best-fitting model varies among alignments, with a Zipf length distribution fitting the vast majority of them. The data underlying this article are available in Github, at https://github.com/elyawy/SpartaSim and https://github.com/elyawy/SpartaPipeline.

Solving a Typical Small Sample Size MRSM Dataset Problem Using a Flexible Hybrid Ensemble Approach for Credibility

Multiresponse surface methodology often involves small data analytics which, statistically, have regression modelling credibility problems. This is worsened by dataset, model selection and solution methodology uncertainties. It is difficult for solution methodologies which select and use single best models per response at simultaneous optimisation to effectively deal with these problems. This paper exploited the fact that model selection criteria choose differently, in a flexible hybrid ensemble system, to generate several solutions for integration and comparison. Mean square prediction error, with bias-variance-covariance decomposition values, was computed and analysed at simultaneous optimisation. Results suggest that the credibility of the final solution is enhanced when working with multiple models, solution methodologies and results. However, the results do not show any significance of small sample size correction to model selection criteria and analysis of bias-variance-covariance decompositions at simultaneous optimisation does not encourage dependence on theoretical optimality for best results.

Calibration, validation, and selection of hydrostatic testing-based remaining useful life prediction models for polyethylene pipes

A novel methodology for model selection among competing models for remaining useful life (RUL) prediction is developed in this paper. Due to the long durability of polyethylene (PE) pipes under normal conditions, life data under normal operating conditions is not available for model validation and selection. Physics-based, regression-based, and hybrid models for RUL prediction are available in the literature based on experimental data under accelerated test conditions. These models need to be evaluated for prediction performance under normal conditions, but in the absence of data under normal conditions. A model consistency-based metric for selecting the best model in the absence of data under normal conditions is proposed in this paper. The consistency-based metric evaluates the predictive consistency of the various probabilistic RUL prediction models over the estimated or known probability distribution of the normal operating conditions. A two-step approach for model selection is proposed: first, validation evaluation using empirical data, and second, consistency evaluation under likely operating conditions. The proposed model selection methodology is demonstrated using five candidate models for RUL prediction of PE pipes based on accelerated hydrostatic testing data. Bayesian inference is used to calibrate these models with empirical data, and its benefit over the least squares approach recommended in ISO 9080 is demonstrated. Further, the proposed two-step model selection methodology is compared against traditional model selection methods based on goodness of fit, model complexity and information theoretic metrics. It is seen that the proposed additional consistency criterion is successful in selecting the best model compared to existing methods that are unable to distinguish between the different available models.

An integrated system to significant wave height prediction: Combining feature engineering, multi-criteria decision making, and hybrid kernel density estimation

Accurate prediction of significant wave height is paramount for the effective design, operation, and maintenance of wave energy converters. However, current research falls short in achieving precise and stable point predictions, along with comprehensive uncertainty analysis of significant wave height. To address this gap, this study presents a comprehensive significant wave height combined prediction system. This integrated system encompasses outlier detection utilizing Autoencoders, sophisticated feature engineering, a multi-criteria decision-based model selection methodology, a multi-objective homogeneous nuclear molecular optimization algorithm, and a hybrid kernel density estimation technique. To tackle the critical issue of model selection within ensemble prediction, we introduce a multi-criteria compromise solution ranking algorithm known as VIKOR for the selection of sub-models. Additionally, a novel multi-objective homogeneous nuclear molecular optimization algorithm is proposed, which incorporates joint opposing selection and an elite retention strategy to effectively manage multiple objectives simultaneously, yielding Pareto optimal solutions for combining weights. Furthermore, a hybrid kernel density estimation approach is developed, surpassing previous methods reliant on a single kernel function and fixed bandwidth, thereby achieving a more precise fit to the distribution of wave height data. The effectiveness of the proposed model is rigorously evaluated using wave height datasets from three distinct locations. The experimental results convincingly demonstrate that the significant wave height combined prediction system outperforms existing solutions, excelling in both point and interval predictions of significant wave height.

Reinforcement learning and approximate Bayesian computation (RL-ABC) for model selection and parameter calibration of time-varying systems

This paper extends the recently developed methodology for model selection and parameter identification called RL-ABC (Ritto et al., 2022) (reinforced learning and approximate Bayesian computation) to time-varying systems. To tackle slowly-varying systems and detect abrupt changes, new features are proposed. (1) The probability of sampling the worst model has now a lower bound; because it cannot disappear, once it might be useful in the future as the system evolves. (2) A memory term (sliding window) is introduced such that past data can be forgotten whilst updating the reward; which might be useful depending on how fast the system changes. (3) The algorithm detects a change in the system by monitoring the models’ acceptance; a significant drop in acceptance indicates a change. If the system changes the algorithm is reset: new parameter ranges are computed and the rewards are restarted. To test the proposed strategy, new experimental data is obtained from a test rig with non-linear restoring force characteristics. The amplitude of the dynamical experiment is obtained with the control-based continuation strategy varying the excitation amplitude, and three Duffing-like models are used to represent the system. The results are consistent, and the strategy is able to detect changes and update parameter estimation and model predictions.

Modelling circular time series

Circular variables often play an important role in the construction of models for analysing and forecasting the consequences of climate change and its impact on the environment. Such variables pose special problems for time series modelling. This article shows how the score-driven approach, developed primarily in econometrics, provides a natural solution to the difficulties and leads to a coherent and unified methodology for estimation, model selection and testing. The new methods are illustrated with data on wind direction.

Machine Learning Classification Model Comparison

Machine learning models are boosting Artificial Intelligence applications in many domains, such as automotive, finance and health care. This is mainly due to their advantage, in terms of predictive accuracy, with respect to classic statistical models. However, machine learning models are much less explainable: less transparent, less interpretable. This paper proposes to improve machine learning models, by proposing a model selection methodology, based on Lorenz Zonoids, which allows to compare them in terms of predictive accuracy significant gains, leading to a selected model which maintains accuracy while improving explainability. We illustrate our proposal by means of simulated datasets and of a real credit scoring problem. The analysis of the former shows that the proposal improves alternative methods, based on the AUROC. The analysis of the latter shows that the proposal leads to models made up of two/three relevant variables that measure the profitability and the financial leverage of the companies asking for credit.

Reinforcement learning and approximate Bayesian computation for model selection and parameter calibration applied to a nonlinear dynamical system

In the context of digital twins and integration of physics-based models with machine learning tools, this paper proposes a new methodology for model selection and parameter identification. It combines (i) reinforcement learning (RL) for model selection through a Thompson-like sampling with (ii) approximate Bayesian computation (ABC) for parameter identification and uncertainty quantification. These two methods are applied together to a nonlinear mechanical oscillator with periodic forcing. Experimental data are used in the analysis and two different nonlinear models are tested. The initial Beta distribution that represents the likelihood of the model is updated depending on how successful the model is at reproducing the reference data (reinforcement learning strategy). At the same time, the prior distribution of the model parameters is updated using a likelihood-free strategy (ABC). In the end, the rewards and the posterior distribution of the parameters of each model are obtained. The results show that the combined methodology (RL-ABC) is promising for model selection from bifurcation diagrams. Prior parameter distribution was successfully updated, correlations between parameters were found, probabilistic envelopes of the posterior model are consistent with the available data, the most rewarded model was selected, and the reinforcing strategy allows to speed up the selection process.

Modelling soil erosion by water under future climate change: Addressing methodological gaps

Soil erosion by water from arable land poses a serious threat to on-field agricultural productivity and the wider environment through off-site damage. Multiple studies show that climate change will worsen the impacts of soil erosion in various regions. However, these studies are limited by (1) the lack of any thorough evaluation process in applying climate scenarios to drive soil erosion models, and (2) the failure to consider the role of changing land use under future climate change, despite the evidence that it is more important than rainfall changes in driving increased erosion. Using the WEPP soil erosion model, these methodological gaps are addressed in this study for a small catchment in Belgium that is both heavily impacted by soil erosion and boasting an extensive array of mitigation measures. We develop a novel and comprehensive methodology to rigorously and efficiently select suitable climate models specifically for simulating soil erosion by water, and examine the impact of a range of environmentally and economically viable land use choices on soil erosion. The main findings reveal that our climate model selection methodology is successful in generating the widest range of likely future scenarios from a small number of models, compared with other selection methods. Our novel methodology reveals that the magnitude and frequency of soil erosion events will increase considerably under the mean of all scenarios between 2041 and 2100 with existing land management. Winter wheat represents the most economically and environmentally viable land use choice to effectively mitigate future soil erosion when compared to other land use alternatives under the full range of likely future climate scenarios. This research illuminates the importance of carefully tuned climate model selection and land use changes for modelling future soil erosion by water so the best- and worst-case scenarios can be adequately prepared for under a changing climate.

On Order Selection of Arima Models for Stationary Normal and Non-Normal Data Structures

The study is aimed at identifying the orders of time series models in stationary and non-stationary non-normal data from different distributions; with a view to determining the best Autoregressive/ Moving Average orders from different time series, ARIMA models were considered for different underline distributions of data. The data is generated under normal uniform and exponential distribution using a second-order autoregressive model. Data were generated in two forms, these are, when stationarity is observed and when it is violated. Each case of data simulated is fitted to different models and the values of AIC, BIC, HQIC, and FPE are computed. The effect of different levels of parameters (0.3, 0.6, and -0.3, -0.6) at the sample size of 20, 40, 60, 80, 100, 120, 140, 160, 180, and 200 which we considered to represent moderate and large sample sizes respectively on the simulated data from the stationary normal and non-normal data. We concluded in general that the selections of the order for the models considered in this study are tied more to the underlying distribution of the series in relation to the Stationarity and non-Stationarity of the series as it will lead to the identification of the proper model. Since the selection are almost identical in the stationary and non-stationary from both normal and non-normal data structure but varies with the variation in the distribution of the series. And it was also observed that for the ARIMA models the order stocked to the principle of parsimony i.e. models with lower-order selected at most of the sample sizes considered. The need to develop a methodology for model selection that combines both objective and subjective techniques is strongly recommended.

Voting Classification-Based Diabetes Mellitus Prediction Using Hypertuned Machine-Learning Techniques

Diabetes mellitus is a hyperglycemia-like chronic condition that is a troublesome disease. It is estimated that, according to the growing morbidity, by 2040, the world will cross 642 million diabetic patients. This means that each one of the ten adults will be diabetes-affected. Diabetes can also lead to other illnesses such as heart attacks, kidney damage, and even blindness. The prediction of diabetes in advance motivates us to develop a machine learning-based model. A dataset was obtained from the online repository for this work. The obtained dataset was imbalanced. An imbalanced dataset presents a challenge that is needed to be balanced for prediction using multiple machine learning like Tomek and SMOTE. These techniques remove necessary outliers that are incomplete in the provided dataset. These outliers are also managed using the IQR method. Additionally, this research employed a two-stage model selection methodology. In the first stage, logistic regression, Support Vector Machine, k-nearest neighbors, gradient boost, Naive Bayes, and Random Forests were applied to determine the efficiency of prediction based on patients’ preconditioning. At this stage, Random Forest was found to be the best with an accuracy of 80.7% after applying SMOTE oversampling technique to balance the dataset. In the second stage, three better-performing models were used by utilizing a voting algorithm. The results were encouraging, and the model obtained 82.0% accuracy with the default dataset and 81.7% accuracy with the balanced dataset. Naive Bayes Theorem, Gradient Boosting Classifier, and Random Forest were used as inputs to the voting algorithm.

Optimal Neighborhood Selection for AR-ARCH Random Fields with Application to Mortality

This article proposes an optimal and robust methodology for model selection. The model of interest is a parsimonious alternative framework for modeling the stochastic dynamics of mortality improvement rates introduced recently in the literature. The approach models mortality improvements using a random field specification with a given causal structure instead of the commonly used factor-based decomposition framework. It captures some well-documented stylized facts of mortality behavior including: dependencies among adjacent cohorts, the cohort effects, cross-generation correlations, and the conditional heteroskedasticity of mortality. Such a class of models is a generalization of the now widely used AR-ARCH models for univariate processes. A the framework is general, it was investigated and illustrated a simple variant called the three-level memory model. However, it is not clear which is the best parameterization to use for specific mortality uses. In this paper, we investigate the optimal model choice and parameter selection among potential and candidate models. More formally, we propose a methodology well-suited to such a random field able to select thebest model in the sense that the model is not only correct but also most economical among all thecorrectmodels. Formally, we show that a criterion based on a penalization of the log-likelihood, e.g., the using of the Bayesian Information Criterion, is consistent. Finally, we investigate the methodology based on Monte-Carlo experiments as well as real-world datasets.

Automated classification of cytogenetic abnormalities in hematolymphoid neoplasms.

Algorithms for classifying chromosomes, like convolutional deep neural networks (CNNs), show promise to augment cytogeneticists' workflows; however, a critical limitation is their inability to accurately classify various structural chromosomal abnormalities. In hematopathology, recurrent structural cytogenetic abnormalities herald diagnostic, prognostic and therapeutic implications, but are laborious for expert cytogeneticists to identify. Non-recurrent cytogenetic abnormalities also occur frequently cancerous cells. Here, we demonstrate the feasibility of using CNNs to accurately classify many recurrent cytogenetic abnormalities while being able to reliably detect non-recurrent, spurious abnormal chromosomes, as well as provide insights into dataset assembly, model selection and training methodology that improve overall generalizability and performance for chromosome classification. Our top-performing model achieved a mean weighted F1 score of 96.86% on the validation set and 94.03% on the test set. Gradient class activation maps indicated that our model learned biologically meaningful feature maps, reinforcing the clinical utility of our proposed approach. Altogether, this work: proposes a new dataset framework for training chromosome classifiers for use in a clinical environment, reveals that residual CNNs and cyclical learning rates confer superior performance, and demonstrates the feasibility of using this approach to automatically screen for many recurrent cytogenetic abnormalities while adeptly classifying non-recurrent abnormal chromosomes. Software is freely available at https://github.com/DaehwanKimLab/Chromosome-ReAd. The data underlying this article cannot be shared publicly due to it being protected patient information. Supplementary data are available at Bioinformatics online.

ON TIME SERIES MODEL ORDER SELECTION OF NON-STATIONARY AND NON-NORMAL DATA STRUCTURE

The study is aimed at identifying the orders of Time Series Models in Non-Stationarity Non-normal data structure from Uniform distributions with a view to determining the best Autoregressive/ Moving Average orders from time series models (ARMA and ARIMA) under different underline distributions when Non-Stationarity assumption is assumed. The data is generated from Autoregressive (AR) linear of second orders of general classes of Autoregressive functions. The generation of the data used for this simulation study is non-stationary cases and nonnormal. The data were simulated for both response variables and error terms from non-normal Distribution. The result shows that the values of the penalty function: AIC, BIC, HQIC and FPE of the order selection increases with increase in sample size but decreases with increasing order. It was observed that at both lower (20, 40) and larger (160,180 and 200) sample sizes, models with smallest orders. Similarly, the selection process is tied to the principle of parsimony i.e smaller orders are selected, but vary with the variation in the distribution of the series. The study recommends the need to develop a methodology for model selection combining objective and subjective techniques.

Accurate Surface Condition Classification of High Voltage Insulators based on Deep Convolutional Neural Networks

Outdoor insulators in high voltage power lines serve as electrical insulation barriers and mechanical supports for live conductors. They are exposed to environmental contaminants and physical deterioration or damage. Hence, polluted insulator analysis is a fundamental concern for proper power system operation. This study harvests a comprehensive insulator surface dataset composed of 4500 images under different surface conditions: clean surface, clean surface with the water droplets, contaminated surfaces with the soil and cement, as well as a wet surface, which is mixed with the soil and cement contaminants. Convolutional neural networks (CNNs) are employed, and a systematic model selection methodology is introduced to construct an accurate classifier of insulator surface conditions while taking into consideration the potential implementation of the constructed CNN in resource-limited embedded devices. The results show the proposed model complexity reduction technique leads to a lighter architecture by a factor of ~3 at the expense of a slight reduction of 6.5% in classification accuracy.

REGRESSION ANALYSIS OF THE ENERGY PRODUCED IN COGENERATION AND SUPPLIED TO DISTRICT HEATING SYSTEMS

Currently, the district heating systems (DHSs) are intensively promoted both nationally and globally. The advantages of using these systems in urban areas compared to individual heating systems are practically indisputable. Still, it is essential that the calculation underlying the assessment of the economic profitability of the projects to connect the new heat consumers to DHS be made correctly by taking into account all the necessary investments and the benefits obtained from the additional amount of energy sale. The article presents the methodology for the optimal regression model selection that can be applied to predict the additional electricity produced in cogeneration mode in case of the new heat consumer connection to the DHS, based on the actual data of the thermal and electrical energy supplied to network from a CHP in the Republic of Moldova. At the same time, it is demonstrated that between the thermal energy supplied to a new consumer connected to DHS and the additional electricity produced in the cogeneration mode, there is a direct and linear correlation.