Bayesian Optimized Research Articles

Short-Term Wind Power Prediction Approach Based On Bayesian Optimization and Ensemble Learning

In wind energy studies, predicting the short-term energy generation amount for wind power plants and determining the production offer to be placed on the market play an important role. In this study an hourly short-term wind power estimation of a wind turbine located in Turkey with an installed power of 3600 kW has been made. Estimation results were evaluated on a seasonal and annual basis. New hybrid models have been developed for short-term wind power prediction, consisting of Bayesian Optimization (BO), Support Vector Regression (SVR), Gaussian Process Regression (GPR), Decision Tree (DT), stacking, and bagging algorithms. In the proposed prediction approach, it is aimed to reduce prediction errors by combining different regression algorithms with the BO method and ensemble algorithms. Unlike other wind prediction studies, BO was used for the first time in the hyperparameter selection of the regression algorithms selected as the basic learner in the study. Bayesian optimized decision tree (BO-DT) with the lowest error values among the base learners, and Bayesian optimized gaussian process regression (BO-GPR) combined with bagging and stacking. The efficiency of ensemble learning algorithms was measured by the statistical measurement methods Normalized Absolute Mean Error (NMAE), Normalized Root of Mean Squares Error (NRMSE), and determination coefficient (R 2 ). According to the results, the bagging method created with the BO-DT took the annual average NRMSE, NMAE, R 2 criteria of 11.045%, 4.880%, 0.899, respectively, and the model with the best performance was selected in terms of both annual and seasonal results.

An interpretable multiparametric radiomics model for the diagnosis of schizophrenia using magnetic resonance imaging of the corpus callosum

There is a growing need to develop novel strategies for the diagnosis of schizophrenia using neuroimaging biomarkers. We investigated the robustness of the diagnostic model for schizophrenia using radiomic features from T1-weighted and diffusion tensor images of the corpus callosum (CC). A total of 165 participants [86 schizophrenia and 79 healthy controls (HCs)] were allocated to training (N = 115) and test (N = 50) sets. Radiomic features of the CC subregions were extracted from T1-weighted, apparent diffusion coefficient (ADC), and fractional anisotropy (FA) images (N = 1605). Following feature selection, various combinations of classifiers were trained, and Bayesian optimization was adopted in the best performing classifier. Discrimination, calibration, and clinical utility of the model were assessed. An online calculator was constructed to offer the probability of having schizophrenia. SHapley Additive exPlanations (SHAP) was applied to explore the interpretability of the model. We identified 30 radiomic features to differentiate participants with schizophrenia from HCs. The Bayesian optimized model achieved the highest performance, with an area under the curve (AUC), accuracy, sensitivity, and specificity of 0.89 (95% confidence interval: 0.81–0.98), 80.0, 83.3, and 76.9%, respectively, in the test set. The final model offers clinical probability in an online calculator. The model explanation by SHAP suggested that second-order features from the posterior CC were highly associated with the risk of schizophrenia. The multiparametric radiomics model focusing on the CC shows its robustness for the diagnosis of schizophrenia. Radiomic features could be a potential source of biomarkers that support the biomarker-based diagnosis of schizophrenia and improve the understanding of its neurobiology.

A Bayesian Optimized Discriminant Analysis Model for Condition Monitoring of Face Milling Cutter Using Vibration Datasets

Abstract With the advent of industry 4.0, which conceptualizes self-monitoring of rotating machine parts by adopting techniques like data analytics, cloud computing, Internet of things, machine learning (ML), and artificial intelligence. The significant research area in predictive maintenance is tool condition monitoring (TCM) as the tool condition affects the overall machining process and its economics. Lately, machine learning techniques are being used to classify the tool’s condition in operation. These techniques are cost saving and help industries with adopting future-proof solutions for their operations. One such technique called discriminant analysis (DA) must be examined particularly for TCM. Owing to its less-expensive computation and shorter run times, using them in TCM will ensure the effective use of the cutting tool and reduce maintenance times. This article presents a Bayesian optimized discriminant analysis model to classify and monitor the tool condition into three user-defined classes. The data are collected using an in-house designed and developed data acquisition (DAQ) module setup on a Vertical Machining Center (VMC). The hyperparameter tuning has been incorporated using Bayesian optimization search, and the parameter that gives the best model was found out to be “linear,” achieving an accuracy of 93.3%. This study confirms the feasibility of machine learning techniques like DA in the field of TCM and using Bayesian optimization algorithms to fine-tune the model, making it industry ready.

A Bayesian optimized framework for successful application of unscented Kalman filter in parameter identification of MDOF structures

The success of the unscented Kalman filter can be jeopardized if the required initial parameters are not identified carefully. These parameters include the initial guesses and the levels of uncertainty in the target parameters and the process and measurement noise parameters. While a set of appropriate initial target parameters give the unscented Kalman filter a head start, the uncertainty levels and noise parameters set the rate of convergence in the process. Therefore, due to the coupling effect of these parameters, an inclusive approach is desired to maintain the chance of convergence for expensive experimental tests. In this paper, a framework is proposed that, via a virtual emulation prior to the experiment, determines a set of initial conditions to ensure a successful application of the online parameter identification. A Bayesian optimization method is proposed, which considers the level of confidence in the initial guesses for the target parameters to suggest the appropriate noise covariance matrices. The methodology is validated on a five-story shear frame tested on a shake table. The results indicate that, indeed, a trade-off can be made between the robustness of the online updating and the final parameter accuracy.

Bayesian Optimized Monte Carlo Planning

Online solvers for partially observable Markov decision processes have difficulty scaling to problems with large action spaces. Monte Carlo tree search with progressive widening attempts to improve scaling by sampling from the action space to construct a policy search tree. The performance of progressive widening search is dependent upon the action sampling policy, often requiring problem-specific samplers. In this work, we present a general method for efficient action sampling based on Bayesian optimization. The proposed method uses a Gaussian process to model a belief over the action-value function and selects the action that will maximize the expected improvement in the optimal action value. We implement the proposed approach in a new online tree search algorithm called Bayesian Optimized Monte Carlo Planning (BOMCP). Several experiments show that BOMCP is better able to scale to large action space POMDPs than existing state-of-the-art tree search solvers.

Fast and Efficient High-Sigma Yield Analysis and Optimization Using Kernel Density Estimation on a Bayesian Optimized Failure Rate Model

With ever-increasing transistor density in nanoscale-integrated circuits, the impact of process variations on circuit performance and chip yield becomes dominant. To prevent failures in the field, simulation-based circuit optimization tools are performed during design time as a countermeasure. However, the efficiency of these tools requires accurate modeling of failure probabilities. Especially, for high-sigma problems such as yield estimation of memory cells, which require very high production yield, the failure rate assessment must be highly accurate. Importance sampling (IS) methods are deployed in this context to uncover very rare failure events, which cannot be revealed by standard Monte Carlo methods. Besides a highly accurate yield prediction model, the limited time budget of the simulation-based analysis tools has to be taken into account. Especially in conjunction with yield optimization techniques, the required number of circuit simulations for the failure rate estimation has to be substantially reduced. In this article, we propose a yield optimization method, which is based on a Bayesian optimization (BO) failure rate estimation technique for high-sigma yield extraction. The BO-based IS method is combined with a kernel density estimator for finding the most probable failure events, which have significant contribution to the chip yield. By integration into a global optimization framework, we show how the proposed yield optimization method can be applied to high-sigma yield optimization problems, such as memory cells. The experimental results indicate that the proposed method consumes only 5% circuit simulations to achieve the same optimization effect as the state-of-the-art yield optimizer techniques.

Deep‐features with Bayesian optimized classifiers for the breast cancer diagnosis

AbstractTo improve the state‐of‐the‐art works in breast cancer diagnosis, this paper proposes ensemble‐based deep‐transfer learning with classifiers namely optimizable K‐nearest neighbors, optimizable naïve Bayes, optimizable ensemble, and optimizable support vector machine algorithms for the auto‐feature extraction, detection, and severity classification of input mammograms. The hyperparameters of these classification algorithms are optimized by using the Bayesian optimization technique. The extracted robust features are normalized and then classified using the Bayesian optimized classifiers. Thus, the work throws a light of research on detecting whether the input mammogram is normal or abnormal. Afterward, it further focuses on the severity classification of abnormalities that is, benign or malignant. The aforesaid algorithms are trained and tested for the three‐class classification problem, thus achieving a maximum performance using the Bayesian optimized SVM algorithm applied with ResNet18 deep features providing classification accuracy of 99.689% for MIAS and 98.883% for INbreast dataset.

Rainfall Induced Landslide Susceptibility Mapping Based on Bayesian Optimized Random Forest and Gradient Boosting Decision Tree Models—A Case Study of Shuicheng County, China

Among the most frequent and dangerous natural hazards, landslides often result in huge casualties and economic losses. Landslide susceptibility mapping (LSM) is an excellent approach for protecting and reducing the risks by landslides. This study aims to explore the performance of Bayesian optimization (BO) in the random forest (RF) and gradient boosting decision tree (GBDT) model for LSM and applied in Shuicheng County, China. Multiple data sources are used to obtain 17 conditioning factors of landslides, Borderline-SMOTE and Randomundersample methods are combined to solve the imbalanced sample problem. RF and GBDT models before and after BO are adopted to calculate the susceptibility value of landslides and produce LSMs and these models were compared and evaluated using multiple validation approach. The results demonstrated that the models we proposed all have high enough model accuracy to be applied to produce LSM, the performance of the RF is better than the GBDT model without BO, while after adopting the Bayesian optimized hyperparameters, the prediction accuracy of the RF and GBDT models is improved by 1% and 7%, respectively and the Bayesian optimized GBDT model is the best for LSM in this four models. In summary, the Bayesian optimized RF and GBDT models, especially the GBDT model we proposed for landslide susceptibility assessment and LSM construction has a very good application performance and development prospects.

Acoustic-Based Engine Fault Diagnosis Using WPT, PCA and Bayesian Optimization

Engine fault diagnosis aims to assist engineers in undertaking vehicle maintenance in an efficient manner. This paper presents an automatic model and hyperparameter selection scheme for engine combustion fault classification, using acoustic signals captured from cylinder heads of the engine. Wavelet Packet Transform (WPT) is utilized for time–frequency analysis, and statistical features are extracted from both high- and low-level WPT coefficients. Then, the extracted features are used to compare three models: (i) standard classification model; (ii) Bayesian optimization for automatic model and hyperparameters selection; and (iii) Principle Component Analysis (PCA) for feature space dimensionality reduction combined with Bayesian optimization. The latter two models both demonstrated improved accuracy and the other performance metrics compared to the standard model. Moreover, with similar accuracy level, PCA with Bayesian optimized model achieved around 20% less total evaluation time and 8–19% less testing time, compared to the second model, for all fault conditions, which thus shows a promising solution for further development in real-time engine fault diagnosis.

Automatic detection of intracranial aneurysms in 3D-DSA based on a Bayesian optimized filter

BackgroundIntracranial aneurysm is a common type of cerebrovascular disease with a risk of devastating subarachnoid hemorrhage if it is ruptured. Accurate computer-aided detection of aneurysms can help doctors improve the diagnostic accuracy, and it is very helpful in reducing the risk of subarachnoid hemorrhage. Aneurysms are detected in 2D or 3D images from different modalities. 3D images can provide more vascular information than 2D images, and it is more difficult to detect. The detection performance of 2D images is related to the angle of view; it may take several angles to determine the aneurysm. As the gold standard for the diagnosis of vascular diseases, the detection on digital subtraction angiography (DSA) has more clinical value than other modalities. In this study, we proposed an adaptive multiscale filter to detect intracranial aneurysms on 3D-DSA.MethodsAdaptive aneurysm detection consists of three parts. The first part is a filter based on Hessian matrix eigenvalues, whose parameters are automatically obtained by Bayesian optimization. The second part is aneurysm extraction based on region growth and adaptive thresholding. The third part is the iterative detection strategy for multiple aneurysms.ResultsThe proposed method was quantitatively evaluated on data sets of 145 patients. The results showed a detection precision of 94.6%, and a sensitivity of 96.4% with a false-positive rate of 6.2%. Among aneurysms smaller than 5 mm, 93.9% were found. Compared with aneurysm detection on 2D-DSA, automatic detection on 3D-DSA can effectively reduce the misdiagnosis rate and obtain more accurate detection results. Compared with other modalities detection, we also get similar or better detection performance.ConclusionsThe experimental results show that the proposed method is stable and reliable for aneurysm detection, which provides an option for doctors to accurately diagnose aneurysms.

Bayesian Optimized Echo State Network Applied to Short-Term Load Forecasting

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

Bayesian optimized deep convolutional network for bearing diagnosis

The successful diagnosis of the faulty signal in rolling element bearings relies on the accurate evaluation of the early fault present within the components of bearings. Because of system imperfection and interference between the data acquisition devices, the fault signal is heavily masked by noise. Hence, to extract the fault information, signal processing techniques are widely used in bearing diagnosis. Although numerous research have dedicated on finding representative features to indicate the damage of the bearing elements, the correlation between defect size and acquired vibration signal has not been properly established. In the recent few years, deep learning has been widely used in bearing diagnosis. In general, the unprocessed signal is directly input into the deep learning model and the neural network extracts useful features during the optimization process. Until now, the selection of features from signals is arbitrary which does not yield much insights into the diagnosis and prognosis. In addition, the features could contain noise information, which could possibly deteriorate diagnostic or prognostic results, while the effect of using preprocessed data has not been fully explored. In this paper, we present an innovative diagnosis model using the deep convolutional network with Bayesian optimization to diagnose the defect severity of bearings. The acquired signal is initially processed using the complementary ensemble empirical mode decomposition method to extract the frequency band containing the fault signature. An experimental based defect size estimation equation is implemented to calculate the defect size based on the signal and experimental setup. After the estimated defect size is obtained, the deep convolutional neural network is implemented to categorize the defect severity. The parameters of the network are optimized by the Bayesian algorithm. The proposed algorithm can be used for diagnosis of the health condition of various rotating machinery.

A new W-SVM kernel combining PSO-neural network transformed vector and Bayesian optimized SVM in GDP forecasting

Considering that in the literature there is a very limited number of studies proposing new SVM kernels especially in regression problems, the scope of this research is to investigate the development of a novel Support Vector Machine Kernel. The proposed new W-SVM (Weighted-SVM) kernel was developed by applying a suitably transformed weight vector derived from particle swarm optimized neural networks in order to satisfy the kernel conditions of Mercer’s theorem and then incorporated to a Bayesian Optimized (BO) kernel for building the new proposed W-SVM kernel. The proposed SVM kernel was applied in Gross Domestic Product growth forecasting. The new kernel has led to significantly improved forecasting results compared to all the other conventional ANN, SVM, and optimized BO-SVM, PSO-ANN machine learning models.

Assessment of agricultural energy consumption of Turkey by MLR and Bayesian optimized SVR and GPR models

AbstractAgricultural productivity highly depends on the cost of energy required for cultivation. Thus prior knowledge of energy consumption is an important step for energy planning and policy development in agriculture. The aim of the present study is to evaluate the application potential of multiple linear regression (MLR) and machine learning tools such as support vector regression (SVR) and Gaussian process regression (GPR) to forecast the agricultural energy consumption of Turkey. In the development of the models, widespread indicators such as agricultural value‐added, total arable land, gross domestic product share of agriculture, and population data were used as input parameters. Twenty‐eight‐year historical data from 1990 to 2017 were utilized for the training and testing stages of the models. A Bayesian optimization method was applied to improve the prediction capability of SVR and GPR models. The performance of the models was measured by various statistical tools. The results indicated that the Bayesian optimized GPR (BGPR) model with exponential kernel function showed a superior prediction capability over MLR and Bayesian optimized SVR model. The root mean square error, mean absolute deviation, mean absolute percentage error, and coefficient of determination (R2) values for the BGPR model were determined as 0.0022, 0.0005, 0.2041, and 0.9999 in the training phase and 0.0452, 0.0310, 7.7152, and 0.9677 in the testing phase, respectively. As a result, it can be concluded that the proposed BGPR model is an efficient technique and has the potential to predict agricultural energy consumption with high accuracy.

Bayesian Optimized Mixture Importance Sampling for High-Sigma Failure Rate Estimation

In many application domains, in particular automotives, guaranteeing a very low failure rate is crucial to meet functional and safety standards. Especially, reliable operation of memory components such as SRAM cells is of essential importance. Due to aggressive technology downscaling, process and runtime variations significantly impact manufacturing yield as well as functionality. For this reason, a thorough memory failure rate assessment is imperative for correct circuit operation and yield improvement. In this regard, Monte Carlo (MC) simulations have been used as the conventional method to estimate the variability induced failure rate of memory components. However, MC methods become infeasible when estimating rare events such as high-sigma failure rates. To this end, importance sampling (IS) methods have been proposed which reduce the number of required simulations substantially. However, existing methods still suffer from inaccuracies and high computational efforts, in particular for high-sigma problems. In this article, we fill this gap by presenting an efficient mixture IS approach based on Bayesian optimization, which deploys a surface model of the objective function to find the most probable failure points. Its advantages include constant complexity independent of the dimensions of design space, the potential to find the global extrema, and the higher trustworthiness of the estimated failure rate by accurately exploring the design space. The approach is evaluated on a 6T-SRAM cell as well as a master–slave latch based on a 28-nm FDSOI process. The results show an improvement in accuracy, resulting in up to $63\times $ better accuracy in estimating failure rates compared to the best state-of-the-art solutions on a 28-nm technology node.

Bayesian Optimized Deep Convolutional Network for Electrochemical Drilling Process

Electrochemical machining is a promising non-traditional manufacturing process to make high-quality parts. The benefits of minimal thermally and mechanically induced stresses, free of burr, and a low surface roughness are appealing for industry and research institutes. However, the combined chemical reaction, electric field, fluid mechanics, and material properties involve a significant number of independent parameters which are difficult to analyze in order to draw comprehensive conclusions. To our current knowledge, process responses such as the material removal rate, optimal feed rate, and cutting profile cannot be represented accurately by analytical solutions. In recent years, deep learning has had tremendous success in analyzing sophisticated systems. The improved computation efficiency and reduced size of the training dataset required for deep learning have enabled various prediction models in the manufacturing industry. In this paper, a new approach is developed using the deep convolutional network with the Bayesian optimization algorithm to predict the diameters of the drilled hole from an electrochemical machining process. The Keras application programming interface (API) was used to build the deep convolutional network; the feed rate, pulse-on time, and voltage were used as input parameters to provide a fair comparison with a neural network from previous research. Random dropout layers were added to prevent overfitting of the network. Instead of tuning the network parameter by trial and error, the Bayesian parameter optimization algorithm was implemented to find the optimal set of parameters of the deep convolutional network that yields the minimum mean square error. The proposed algorithm was compared with a previously developed neural network with partially embedded physical knowledge. Improved training speed and accuracy were observed in comparison with the traditional neural network. The prediction model using the proposed deep learning algorithm demonstrated better prediction accuracy and provided a more systematic way to select the hyperparameter for the deep convolutional network.

Classification and mapping of sound sources in local urban streets through AudioSet data and Bayesian optimized Neural Networks

Abstract Deep learning (DL) methods have provided several breakthroughs in conventional data analysis techniques, especially with image and audio datasets. Rapid assessment and large-scale quantification of environmental attributes have been possible through such models. This study focuses on the creation of Artificial Neural Networks (ANN) and Recurrent Neural Networks (RNN) based models to classify sound sources from manually collected sound clips in local streets. A subset of an openly available AudioSet data is used to train and evaluate the model against the common sound classes present in the urban streets. The collection of audio data is done at random locations in the selected study area of 0.2 sq. km. The audio clips are further classified according to the extent of anthropogenic (mainly traffic), natural and human-based sounds present in particular locations. Rather than the manual tuning of model hyperparameters, the study utilizes Bayesian Optimization to obtain hyperparameter values of Neural Network models. The optimized models produce an overall accuracy of 89 percent and 60 percent on the evaluation set for three and fifteen-class model respectively. The model detections are mapped in the study area with the help of the Inverse Distance Weighted (IDW) spatial interpolation method.

Predicting clinical outcomes from large scale cancer genomic profiles with deep survival models

Translating the vast data generated by genomic platforms into accurate predictions of clinical outcomes is a fundamental challenge in genomic medicine. Many prediction methods face limitations in learning from the high-dimensional profiles generated by these platforms, and rely on experts to hand-select a small number of features for training prediction models. In this paper, we demonstrate how deep learning and Bayesian optimization methods that have been remarkably successful in general high-dimensional prediction tasks can be adapted to the problem of predicting cancer outcomes. We perform an extensive comparison of Bayesian optimized deep survival models and other state of the art machine learning methods for survival analysis, and describe a framework for interpreting deep survival models using a risk backpropagation technique. Finally, we illustrate that deep survival models can successfully transfer information across diseases to improve prognostic accuracy. We provide an open-source software implementation of this framework called SurvivalNet that enables automatic training, evaluation and interpretation of deep survival models.