Bayesian Hyperparameter Optimization Research Articles

A boosting ensemble learning based hybrid light gradient boosting machine and extreme gradient boosting model for predicting house prices

AbstractThe implementation of tree‐ensemble models has become increasingly essential in solving classification and prediction problems. Boosting ensemble techniques have been widely used as individual machine learning algorithms in predicting house prices. One of the techniques is LGBM algorithm that employs leaf wise growth strategy, reduces loss and improves accuracy during training which results in overfitting. However, XGBoost algorithm uses level wise growth strategy which takes time to compute resulting in higher computation time. Nevertheless, XGBoost has a regularization parameter, implements column sampling and weight reduction on new trees which combats overfitting. This study focuses on developing a hybrid LGBM and XGBoost model in order to prevent overfitting through minimizing variance whilst improving accuracy. Bayesian hyperparameter optimization technique is implemented on the base learners in order to find the best combination of hyperparameters. This resulted in reduced variance (overfitting) in the hybrid model since the regularization parameter values were optimized. The hybrid model is compared to LGBM, XGBoost, Adaboost and GBM algorithms to evaluate its performance in giving accurate house price predictions using MSE, MAE and MAPE evaluation metrics. The hybrid LGBM and XGBoost model outperformed the other models with MSE, MAE and MAPE of 0.193, 0.285, and 0.156 respectively.

Data‐driven prediction of the probability of creep–fatigue crack initiation in 316H stainless steel

AbstractStainless steel components in advanced gas‐cooled reactors (AGRs) are susceptible to creep–fatigue cracking at high temperatures. Quantifying the probability of creep–fatigue crack initiation requires probabilistic numerical simulations; these are complex and computationally intensive. Here, we present a data‐driven approach to develop fast probabilistic surrogate models of creep–fatigue crack initiation in 316H stainless steel. We perform a set of Monte Carlo simulations based on the R5V2/3 high temperature assessment procedure and determine the sensitivity of the probability of crack initiation to loads and operating conditions. The data are used to train different supervised machine learning models considering Bayesian hyperparameter optimization. We discuss the relative performance of such models and show that a gradient tree boosting algorithm results in surrogate models with the highest accuracy.

Detection of COVID-19 and its pulmonary stage using Bayesian hyperparameter optimization and deep feature selection methods.

Since the first case of COVID‐19 was reported in December 2019, many studies have been carried out on artificial intelligence for the rapid diagnosis of the disease to support health services. Therefore, in this study, we present a powerful approach to detect COVID‐19 and COVID‐19 findings from computed tomography images using pre‐trained models using two different datasets. COVID‐19, influenza A (H1N1) pneumonia, bacterial pneumonia and healthy lung image classes were used in the first dataset. Consolidation, crazy‐paving pattern, ground‐glass opacity, ground‐glass opacity and consolidation, ground‐glass opacity and nodule classes were used in the second dataset. The study consists of four steps. In the first two steps, distinctive features were extracted from the final layers of the pre‐trained ShuffleNet, GoogLeNet and MobileNetV2 models trained with the datasets. In the next steps, the most relevant features were selected from the models using the Sine–Cosine optimization algorithm. Then, the hyperparameters of the Support Vector Machines were optimized with the Bayesian optimization algorithm and used to reclassify the feature subset that achieved the highest accuracy in the third step. The overall accuracy obtained for the first and second datasets is 99.46% and 99.82%, respectively. Finally, the performance of the results visualized with Occlusion Sensitivity Maps was compared with Gradient‐weighted class activation mapping. The approach proposed in this paper outperformed other methods in detecting COVID‐19 from multiclass viral pneumonia. Moreover, detecting the stages of COVID‐19 in the lungs was an innovative and successful approach.

Attention-LSTM architecture combined with Bayesian hyperparameter optimization for indoor temperature prediction

Accurate prediction of indoor temperature can provide more reference data for indoor thermal comfort assessment and the operational effectiveness of heating, ventilation and air conditioning equipment, making it possible to reduce unnecessary energy consumption while ensuring occupant comfort. This paper introduces a deep learning method to predict indoor air temperature. The aim is to explore the potential of a model combining LSTM with encoder-decoder and attention mechanisms in short-term forecasting and compare it with LSTM models and GRU models. The hyperparameters are optimized by TPE Bayesian optimization to facilitate the determination of various parameters in the deep model. The results show that compared with other commonly used time series prediction algorithms, the model has an advantage in the case of short-term time ahead prediction. The model can accurately predict the change trend of room temperature and maintain stability for a long time. The R-square of the prediction results is more than 0.9. This work has reference significance for the feasibility study of establishing an indoor temperature prediction model.

An Entropy-Based Architecture for Detection of Sepsis in Newborn Cry Diagnostic Systems.

The acoustic characteristics of cries are an exhibition of an infant’s health condition and these characteristics have been acknowledged as indicators for various pathologies. This study focused on the detection of infants suffering from sepsis by developing a simplified design using acoustic features and conventional classifiers. The features for the proposed framework were Mel-frequency Cepstral Coefficients (MFCC), Spectral Entropy Cepstral Coefficients (SENCC) and Spectral Centroid Cepstral Coefficients (SCCC), which were classified through K-nearest Neighborhood (KNN) and Support Vector Machine (SVM) classification methods. The performance of the different combinations of the feature sets was also evaluated based on several measures such as accuracy, F1-score and Matthews Correlation Coefficient (MCC). Bayesian Hyperparameter Optimization (BHPO) was employed to tailor the classifiers uniquely to fit each experiment. The proposed methodology was tested on two datasets of expiratory cries (EXP) and voiced inspiratory cries (INSV). The highest accuracy and F-score were 89.99% and 89.70%, respectively. This framework also implemented a novel feature selection method based on Fuzzy Entropy (FE) as a final experiment. By employing FE, the number of features was reduced by more than 40%, whereas the evaluation measures were not hindered for the EXP dataset and were even enhanced for the INSV dataset. Therefore, it was deduced through these experiments that an entropy-based framework is successful for identifying sepsis in neonates and has the advantage of achieving high performance with conventional machine learning (ML) approaches, which makes it a reliable means for the early diagnosis of sepsis in deprived areas of the world.

Model Optimization Analysis of Customer Churn Prediction Using Machine Learning Algorithms with Focus on Feature Reductions

Currently, Customers are struggling to retain their business in today’s competitive markets. Thus, the issue of customer churn becomes a significant challenge for the industries. In order to achieve this, it is vital to have an efficient churn prediction system. In this paper, we discuss methods for reducing features using PCA, Autoencoders, LDA, T-SNE, and Xgboost. In this paper, a model for predicting light GBM churn is proposed. The model consists of five steps. The first step is to preprocess the data so that missing and corrupt values can be handled and the data can be scaled. Secondly, implementing a comprehensive feature reduction system based on popular algorithms reduces the features and selects the most suitable one. In the third step, light GBM’s hyperparameter is tuned using Bayesian hyperparameter optimization and genetic optimization algorithms. Lastly, interpreting the model and evaluating the impact of the features on model outputs by using the SHAP method, and finally ranking the churners by customer lifetime value. Aside from evaluating and choosing the best feature reduction methods, the proposed method is also evaluated using four famous datasets. It outperforms other ensemble and ML algorithms like AdaBoost, SVM, and decision tree on over seven evaluation metrics: accuracy, area under the curve (AUC), Kappa, Mathews correlation coefficient (MCC), Brier score, F1 score, and EMPC. In light of the evaluation metrics, our model shows a significant improvement in handling imbalanced datasets in churn prediction. Finally, in this paper, interpretability and how the features affect the model’s output are presented by the SHAP method. Then CLV ranking is suggested for better decision-making.

Data driven strength and strain enhancement model for FRP confined concrete using Bayesian optimization

To ensure the safe design of composite system, a precise design-based model for concrete encased in fibre-reinforced polymer (FRP) is required. An enclosure data driven model is developed in this paper to forecast the ultimate state of FRP-confined concrete using Bayesian hyperparameter optimization. For a database of 820 columns with circular cross-section, the suggested model's predictions are compared to those of six empirical models and a non-optimized machine learning regressor of XGBoost. In comparison to the previous models, statistical analysis demonstrates that the improved model (BO-XGB) more precisely forecasts the compressive strength and axial strain of concrete confined with FRP. The results further reveal that for strength and strain increase with lateral confinement, BO-XGB with weight uncertainty offers stable predictions, whereas the other models display turbulent model mistake. In comparison to previous models, the suggested model's excellent accuracy and stability are attributable to its great flexibility and resilience in capturing the impacts of lateral confinement pressure as an interaction between the concrete core and FRP shell.

A Short-Term Wind Power Forecast Method via XGBoost Hyper-Parameters Optimization

The improvement of wind power prediction accuracy is beneficial to the effective utilization of wind energy. An improved XGBoost algorithm via Bayesian hyperparameter optimization (BH-XGBoost method) was proposed in this article, which is employed to forecast the short-term wind power for wind farms. Compared to the XGBoost, SVM, KELM, and LSTM, the results indicate that BH-XGBoost outperforms other methods in all the cases. The BH-XGBoost method could yield a more minor estimated error than the other methods, especially in the cases of wind ramp events caused by extreme weather conditions and low wind speed range. The comparison results led to the recommendation that the BH-XGBoost method is an effective method to forecast the short-term wind power for wind farms.

Bayesian optimization for selecting efficient machine learning regressors to determine bond-slip model of FRP-to-concrete interface

Interfacial bond behaviour among FRP and concrete is a crucial aspect in determining the strengthening performance of FRP. A data-driven approach is proposed to automatically identify the key parameters to quantify the peak interfacial bond stress (τm) with the corresponding interfacial fracture energy (Gf), and consequently the bond-slip (τ-s) model can be determined as well. Hyperparameter optimization for selecting the most efficient machine learning model is conducted to optimize the prediction accuracy of the selected model. Bayesian hyperparameter optimization using Gaussian process is used to construct the probability model of the objective function and use it to choose the most favourable hyperparameters for evaluation in the real objective function. Since Catboost regressor shows the lowest RMSE and the best prediction accuracy, the best combination of parameters for rate of learning, dense layer numbers, nodes number corresponding to each layer, activation function, and dropout rate are tuned for further optimization of its prediction accuracy. To further verify the validity of the model prediction, a refined numerical modelling using LS-DYNA is employed to simulate the interfacial fracture process. Then the defined bond-slip model is fed as input to the contact relationship among FRP and concrete, and the recovered load-slip curves are used to compare with experimental data to verify the prediction accuracy of the proposed method.

Application of a Hybrid Model Based on ICEEMDAN, Bayesian Hyperparameter Optimization GRU and the ARIMA in Nonferrous Metal Price Prediction

Nonferrous metals are the basic materials for national economic development. Accurate and robust price forecasting can effectively reduce investment costs and bring greater economic benefits to enterprises. But the violent fluctuation of nonferrous metal prices and the evolution of irregular cycles make metal price prediction difficult. A nonferrous metal price prediction model based on improved complementary ensemble empirical mode (ICEEMDAN) decomposition, a Bayesian hyperparameter optimization gated recurrent neural network (GRU) and an integrated autoregressive moving average model (ARIMA) is proposed. First, ICEEMDAN is used to decompose the original metal price series, and then fuzzy entropy is applied to judge the complexity of confusion of the decomposed subsequence. An ARIMA model is used to predict the sequence with the minimum fuzzy entropy, and the other subsequences are predicted by the Bayesian hyperparametric optimization GRU in this article. Finally, the prediction results are integrated to obtain the final prediction results of the original price series. To verify the reliability and practicability of the model in this article, different lengths of daily futures price series of zinc, aluminum, copper and gold prices on the London Metal Exchange and the Investing are used as the research data, three experiments utilizing 5 groups of metal price data and 10 comparison models are designed to verify the superiority of the proposed model. The results show that: (i) Bayesian hyperparameter optimization can improve the prediction performance of a single model effectively. The hybrid method with fuzzy entropy can effectively improve the prediction accuracy. (ii) The proposed model outperforms the compared state-of-the art models in three-step metal prices forecasting.

High strength aluminum alloys design via explainable artificial intelligence

Here, we have approached to discover new aluminum (Al) alloys with the assistance of artificial intelligence (A.I.) for the enhanced mechanical property. A high prediction rate of 7xxx series Al alloy was achieved via the Bayesian hyperparameter optimization algorithm. With the guide of A.I.-based recommendation algorithm, new Al alloys were designed that had an excellent combination of strength and ductility with a yield strength (YS) of 712 MPa and elongation (EL) of 19%, exhibiting a homogeneous distribution of nanoscale precipitates hindering dislocation movement during deformation. Adding Mg and Cu was found to be the critical factor that decides the relative ratio of strength and EL. We also demonstrate an explainable A.I. (XAI) system that reveals the relationship between input and output parameters. Our A.I. assistant system can accelerate the search for high-strength Al alloys for both experts and non-experts in the field of Al alloy design.

Hybrid Short-term Load Forecasting Method Based on Empirical Wavelet Transform and Bidirectional Long Short-term Memory Neural Networks

Accurate short-term load forecasting is essential to modern power systems and smart grids. The utility can better implement demand-side management and operate power system stably with a reliable load forecasting system. The load demand contains a variety of different load components, and different loads operate with different frequencies. The conventional load forecasting methods, e.g., linear regression (LR), auto-regressive integrated moving average (ARIMA), deep neural network, ignore the frequency domain and can only use time-domain load demand as inputs. To make full use of both time-domain and frequency-domain features of the load demand, a load forecasting method based on hybrid empirical wavelet transform (EWT) and deep neural network is proposed in this paper. The proposed method first filters noises via wavelet-based denoising technique, and then decomposes the original load demand into several sub-layers to show the frequency features while the time-domain information is preserved as well. Then, a bidirectional long short-term memory (LSTM) method is trained for each sub-layer independently. In order to better tune the hyper-parameters, a Bayesian hyperparameter optimization (BHO) algorithm is adopted in this paper. Three case studies are de-signed to evaluate the performance of the proposed method. From the results, it is found that the proposed method improves the prediction accuracy compared with other load forecasting method.

Bayesian Hyperparameter Optimization and Ensemble Learning for Machine Learning Models on Software Effort Estimation

In recent decades, various software effort estimation (SEE) algorithms have been suggested. Unfortunately, generating high-precision accuracy is still a major challenge in the context of SEE. The use of traditional techniques and parametric approaches is largely inaccurate because they produce biased and subjective accuracy. Meanwhile, none of the machine learning methods performed well. This study applies the AdaBoost ensemble learning method and random forest (RF), on the other hand the Bayesian optimization method is applied to determine the hyperparameters of this model. The PROMISE repository and the ISBSG dataset were used to build the SEE model. The developed model was comprehensively compared with four machine learning methods (classification and regression tree, k-nearest neighbor, multilayer perceptron, and support vector regression) under 3-fold cross validation (CV). It can be seen that the RF method based on AdaBoost ensemble learning and bayesian optimization outperforms this approach. In addition, the AdaBoost-based model assigns a feature importance rating, which makes it a promising tool in software effort prediction.

LightGBM-Based Fault Diagnosis of Rotating Machinery Under Changing Working Conditions Using Modified Recursive Feature Elimination

This article presents an intelligent and accurate framework for fault diagnosis of induction motors using light gradient boosting machine (LightGBM). The proposed framework offers promising generalization ability when the testing data contains new unseen operating conditions unavailable during the training process. After the acquisition of vibration signals and feature extraction in multiple domains, we perform an iterative feature selection (FS) approach by utilizing a modified version of recursive feature elimination (RFE) and the features&#x2019; importance scores obtained by LightGBM. To prevent overfitting and subsequent selection bias, an outer resampling loop encompasses the whole process of our RFE-LightGBM algorithm. Moreover, instead of the conventional resampling methods based on K-fold cross-validation (CV) or leave-one-out CV (LOOCV), we use a new scheme called leave-one-loading-out CV (<i>LOLO-CV</i>). Leveraging <i>LOLO-CV</i>, the proposed FS method identifies the optimal feature subset, making the fault diagnosis robust under changing operating conditions. Then, the final classification is performed with optimal feature subset by training a new LightGBM model with adjusted hyperparameters employing Bayesian optimization. Experimental results from two real case studies show that our proposed fault diagnosis framework achieves accuracies between 98.55% and 100% for various testing scenarios. For example, for the worst-case testing scenario in the bearing dataset of Case Western Reserve University where the no-load data (0hp) is absent during the training process and is only used for testing, the testing accuracy of LightGBM classifier before and after applying the proposed RFE-LightGBM-FS method is 88.04% to 97.23%, respectively. Using the Bayesian hyperparameter optimization further improves the accuracy to 98.55%.

Knowledge-based Implementation of Deep Reinforcement Learning Agents in Assembly

Robotic systems based on Deep Reinforcement Learning have shown great potential to enable assembly systems with higher flexibility and robustness. This paper presents a concept of a Case-Based Reasoning system to automate the implementation process, based on the assumption that similar assembly tasks have similar solutions as used as heuristics in the current manual procedure. For retrieving similar cases a digital description of the assembly task and a method to measure the similarity is introduced. The retrieved cases are then used to warmstart a Bayesian Hyperparameter Optimization. The approach is evaluated on two simulated robot task.

Post-Traumatic Stress Disorder (PTSD) biomarker identification using a deep learning model

Post-traumatic stress disorder (PTSD) is a neuropsychological disorder in which individuals struggle to recover from a traumatizing event. It affects a significant population, including COVID-19 patients, frontline health workers, and war veterans. Given biases associated with self-assessment and diagnosis of PTSD, researchers are actively searching for unbiased biological markers (biomarkers) for predicting PTSD status. The Systems Biology of PTSD Consortium has collected candidate biomarkers for PTSD using molecular and clinical measurements of male war veterans between the ages of 20 and 60. PTSD-positive and negative subjects were separated based on the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) Clinician Administered PTSD Scale (CAPS) scores, derived from structured interviews to measure an individual’s abundance of symptoms, such as re-experiencing, flashbacks, and hyperarousal. CAPS scores higher than 40 were considered PTSD-positive and below 20 were considered negative. We created artificial neural network models to classify PTSD-positive and negative individuals based on metabolomics, micro-RNA (miRNA), protein expression, endocrine markers, and DNA methylation datasets. Model training involved 64 iterations of a Bayesian Hyperparameter Optimization algorithm with 5-fold cross-validation. Each model was calibrated based on cross-validation performance and variance across iterations and then fit to the entire respective dataset (76 PTSD-positive, 76 PTSD-negative). We applied the trained models to an independent validation cohort to assess accuracy on unseen datasets. The top performing datasets from the validation cohort based on classification accuracy were metabolomics (65.2%) and protein expression (61.8%). We anticipate the candidate biomarkers identified in this and future studies will assist with the diagnosis of PTSD.

State‐of‐health estimation and remaining useful life for lithium‐ion battery based on deep learning with Bayesian hyperparameter optimization

Lithium-ion battery state-of-health (SOH) estimation and remaining usable life (RUL) prediction are important for battery prognosis and health management. In this article, a framework-combined deep convolution neural network (DCNN) with double-layer long short-term memory (LSTM) is proposed, which is designed for online health prognosis. Based on the raw data obtained during the constant current charging process, the aging characteristics of the battery can be extracted by DCNN to estimate SOH. The estimation results are then sent to the LSTM for the temporal prediction of the RUL. This framework considers both temporal and spatial characteristics of data, and the powerful spatial feature extraction ability of DCNN and the effectiveness of LSTM for time series problems can ensure the high precision of calculation. At the same time, the hyperparameters of the neural networks, which can highly affect the performance of networks, are obtained by Bayesian optimization to ensure the networks run in the best status. The results show that the proposed method can achieve low root mean square error, which are inferior to 0.0061 and 0.0627 for SOH estimation and RUL prediction, respectively.

Spectral analysis and mapping of blackgrass weed by leveraging machine learning and UAV multispectral imagery

Accurate weed mapping is a prerequisite for site-specific weed management to enable sustainable agriculture. This work aims to analyse (spectrally) and mapping blackgrass weed in wheat fields by integrating Unmanned Aerial Vehicle (UAV), multispectral imagery and machine learning techniques. 18 widely-used Spectral Indices (SIs) are generated from 5 raw spectral bands. Then various feature selection algorithms are adopted to improve model simplicity and empirical interpretability. Random Forest classifier with Bayesian hyperparameter optimization is preferred as the classification algorithm. Image spatial information is also incorporated into the classification map by Guided Filter. The developed framework is illustrated with an experimentation case in a naturally blackgrass infected wheat field in Nottinghamshire, United Kingdom, where multispectral images were captured by RedEdge on-board DJI S-1000 at an altitude of 20 m with a ground spatial resolution of 1.16 cm/pixel. Experimental results show that: (i) a good result (an average precision, recall and accuracy of 93.8%, 93.8%, 93.0%) is achieved by the developed system; (ii) the most discriminating SI is triangular greenness index (TGI) composed of Green-NIR, while wrapper feature selection can not only reduce feature number but also achieve a better result than using all 23 features; (iii) spatial information from Guided filter also helps improve the classification performance and reduce noises.

Improved credit scoring model using XGBoost with Bayesian hyper-parameter optimization

&lt;span&gt;Several credit-scoring models have been developed using ensemble classifiers in order to improve the accuracy of assessment. However, among the ensemble models, little consideration has been focused on the hyper-parameters tuning of base learners, although these are crucial to constructing ensemble models. This study proposes an improved credit scoring model based on the extreme gradient boosting (XGB) classifier using Bayesian hyper-parameters optimization (XGB-BO). The model comprises two steps. Firstly, data pre-processing is utilized to handle missing values and scale the data. Secondly, Bayesian hyper-parameter optimization is applied to tune the hyper-parameters of the XGB classifier and used to train the model. The model is evaluated on four widely public datasets, i.e., the German, Australia, lending club, and Polish datasets. Several state-of-the-art classification algorithms are implemented for predictive comparison with the proposed method. The results of the proposed model showed promising results, with an improvement in accuracy of 4.10%, 3.03%, and 2.76% on the German, lending club, and Australian datasets, respectively. The proposed model outperformed commonly used techniques, e.g., decision tree, support vector machine, neural network, logistic regression, random forest, and bagging, according to the evaluation results. The experimental results confirmed that the XGB-BO model is suitable for assessing the creditworthiness of applicants.&lt;/span&gt;

COVID-19 ResNet: Residual neural network for COVID-19 classification with bayesian data augmentation

COVID-19 is an infectious disease caused by a novel coronavirus called SARS-CoV-2. The first case appeared in December 2019, and until now it still represents a significant challenge to many countries in the world. Accurately detecting positive COVID-19 patients is a crucial step to reduce the spread of the disease, which is characterize by a strong transmission capacity. In this work we implement a Residual Convolutional Neural Network (ResNet) for an automated COVID-19 diagnosis. The implemented ResNet can classify a patient´s Chest-Xray image into COVID-19 positive, pneumonia caused from another virus or bacteria, and healthy. Moreover, to increase the accuracy of the model and overcome the data scarcity of COVID-19 images, a personalized data augmentation strategy using a three-step Bayesian hyperparameter optimization approach is applied to enrich the dataset during the training process. The proposed COVID-19 ResNet achieves a 94% accuracy, 95% recall, and 95% F1-score in test set. Furthermore, we also provide an insight into which data augmentation operations are successful in increasing a CNNs performance when doing medical image classification with COVID-19 CXR.