Algorithm Hyperparameter Optimization Research Articles

DeepQGHO: Quantized Greedy Hyperparameter Optimization in Deep Neural Networks for on-the-Fly Learning

On-the-fly learning is unavoidable for applications that demand instantaneous deep neural network (DNN) training or where transferring data to the central system for training is costly. Hyperparameter optimization plays a significant role in the performance and reliability in deep learning. Many hyperparameter optimization algorithms have been developed for obtaining better validation accuracy in DNNs. Most state-of-the-art hyperparameter optimization techniques are computationally expensive due to the focus on validation accuracy. Therefore, they are unsuitable for on-the-fly learning applications that require faster computation on resource constraint devices (e.g., edge devices). In this paper, we develop a novel greedy approach-based hyperparameter optimization (GHO) algorithm enabling faster computing on edge devices for on-the-fly learning applications. In GHO, we obtain the validation accuracy locally for each of the hyperparameter configurations. The GHO algorithm optimizes each hyperparameter while keeping the others constant in order to converge to the locally optimal solution in the expectation that this choice will lead to a globally optimal solution. We perform an empirical study to compute the performance such as computation time and energy consumption of the GHO and compare it with two state-of-the-art hyperparameter optimization algorithms. We also deploy the GHO algorithm in an edge device to validate the performance of our algorithm. We perform post-training quantization on the GHO algorithm to reduce the inference time. Our GHO algorithm is more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> energy efficient and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> faster than two state-of-the-art hyperparameter optimization techniques on both DNNs and datasets studied in this paper.

Hippo

Hyper-parameter optimization is crucial for pushing the accuracy of a deep learning model to its limits. However, a hyper-parameter optimization job, referred to as a study, involves numerous trials of training a model using different training knobs, and therefore is very computation-heavy, typically taking hours and days to finish. We observe that trials issued from hyper-parameter optimization algorithms often share common hyper-parameter sequence prefixes. Based on this observation, we propose Hippo, a hyper-parameter optimization system that reuses computation across trials to reduce the overall amount of computation significantly. Instead of treating each trial independently as in existing hyper-parameter optimization systems, Hippo breaks down the hyper-parameter sequences into stages and merges common stages to form a tree of stages (a stage tree). Hippo maintains an internal data structure, search plan, to manage the current status and history of a study, and employs a critical path based scheduler to minimize the overall study completion time. Hippo applies to not only single studies but multi-study scenarios as well. Evaluations show that Hippo's stage-based execution strategy outperforms trial-based methods for several models and hyper-parameter optimization algorithms, reducing end-to-end training time by up to 2.76X (3.53x) and GPU-hours by up to 4.81X (6.77x), for single (multiple) studies.

Modified Genetic Algorithm for Feature Selection and Hyper Parameter Optimization: Case of XGBoost in Spam Prediction

Recently, spam on online social networks has attracted attention in the research and business world. Twitter has become the preferred medium to spread spam content. Many research efforts attempted to encounter social networks spam. Twitter brought extra challenges represented by the feature space size, and imbalanced data distributions. Usually, the related research works focus on part of these main challenges or produce black-box models. In this paper, we propose a modified genetic algorithm for simultaneous dimensionality reduction and hyper parameter optimization over imbalanced datasets. The algorithm initialized an eXtreme Gradient Boosting classifier and reduced the features space of tweets dataset; to generate a spam prediction model. The model is validated using a 50 times repeated 10-fold stratified cross-validation, and analyzed using nonparametric statistical tests. The resulted prediction model attains on average 82.32% and 92.67% in terms of geometric mean and accuracy respectively, utilizing less than 10% of the total feature space. The empirical results show that the modified genetic algorithm outperforms <i>Chi</i>² and <i>PCA</i> feature selection methods. In addition, eXtreme Gradient Boosting outperforms many machine learning algorithms, including BERT-based deep learning model, in spam prediction. Furthermore, the proposed approach is applied to SMS spam modeling and compared to related works.

Faster Hyperparameter Optimization via Finding Minimal Regions in Random Forest Regressor

In optimisation problems, it is often more convenient and efficient to approximate a target function with another function, which would be much easier to compute or analyse. Such approximations are called Surrogate Functions. A surrogate function can also be a fairly complex object itself. For example, a Random Forest Regressor (RFR) can be used to approximate target function in Hyperparameter Optimisation processes. Hyperparameter vectors define the feature space, and target algorithm performance is considered the target function. Thus, RFR trained on a set of examples, can approximate algorithm performance, using hyperpa-rameter vectors. However, it is hard to fnd its minima. Thus, “black-box” optimization algorithms are applied for optimization purposes, such as random or local searches. In this paper, we propose a method ShortestForestSearch to search RFR minimum points using branch and bound approach. We conducted several experiments for comparing the algorithm proposed with näıve approaches. ShortestForestSearch demonstrated significant performance improvements and the potential to improve other algorithms which use Random Forest internally as a surrogate function. Based on this, we propose ShortestForestSMAC, a new algorithm for hyperparameter optimization. It is based on the well-known SMAC algorithm and implements idea of ShortestForestSearch for selecting best new candidates on each optimization iteration. Experimental evaluation shows that ShortestForestSMAC outperforms SMAC by achieving similar results in considerably shorter time span.

Hybrid Short-term Load Forecasting Method Based on Empirical Wavelet Transform and Bidirectional Long Short-term Memory Neural Networks

Accurate short-term load forecasting is essential to modern power systems and smart grids. The utility can better implement demand-side management and operate power system stably with a reliable load forecasting system. The load demand contains a variety of different load components, and different loads operate with different frequencies. The conventional load forecasting methods, e.g., linear regression (LR), auto-regressive integrated moving average (ARIMA), deep neural network, ignore the frequency domain and can only use time-domain load demand as inputs. To make full use of both time-domain and frequency-domain features of the load demand, a load forecasting method based on hybrid empirical wavelet transform (EWT) and deep neural network is proposed in this paper. The proposed method first filters noises via wavelet-based denoising technique, and then decomposes the original load demand into several sub-layers to show the frequency features while the time-domain information is preserved as well. Then, a bidirectional long short-term memory (LSTM) method is trained for each sub-layer independently. In order to better tune the hyper-parameters, a Bayesian hyperparameter optimization (BHO) algorithm is adopted in this paper. Three case studies are de-signed to evaluate the performance of the proposed method. From the results, it is found that the proposed method improves the prediction accuracy compared with other load forecasting method.

Model Optimization Analysis of Customer Churn Prediction Using Machine Learning Algorithms with Focus on Feature Reductions

Currently, Customers are struggling to retain their business in today’s competitive markets. Thus, the issue of customer churn becomes a significant challenge for the industries. In order to achieve this, it is vital to have an efficient churn prediction system. In this paper, we discuss methods for reducing features using PCA, Autoencoders, LDA, T‐SNE, and Xgboost. In this paper, a model for predicting light GBM churn is proposed. The model consists of five steps. The first step is to preprocess the data so that missing and corrupt values can be handled and the data can be scaled. Secondly, implementing a comprehensive feature reduction system based on popular algorithms reduces the features and selects the most suitable one. In the third step, light GBM’s hyperparameter is tuned using Bayesian hyperparameter optimization and genetic optimization algorithms. Lastly, interpreting the model and evaluating the impact of the features on model outputs by using the SHAP method, and finally ranking the churners by customer lifetime value. Aside from evaluating and choosing the best feature reduction methods, the proposed method is also evaluated using four famous datasets. It outperforms other ensemble and ML algorithms like AdaBoost, SVM, and decision tree on over seven evaluation metrics: accuracy, area under the curve (AUC), Kappa, Mathews correlation coefficient (MCC), Brier score, F1 score, and EMPC. In light of the evaluation metrics, our model shows a significant improvement in handling imbalanced datasets in churn prediction. Finally, in this paper, interpretability and how the features affect the model’s output are presented by the SHAP method. Then CLV ranking is suggested for better decision‐making.

Hyperparameter Tuned Bidirectional Gated Recurrent Neural Network for Weather Forecasting

Weather forecasting is primarily related to the prediction of weather conditions that becomes highly important in diverse applications like drought discovery, severe weather forecast, climate monitoring, agriculture, aviation, telecommunication, etc. Data-driven computer modelling with Artificial Neural Networks (ANN) can be used to solve non-linear problems. Presently, Deep Learning (DL) based weather forecasting models can be designed to accomplish reasonable predictive performance. In this aspect, this study presents a Hyper Parameter Tuned Bidirectional Gated Recurrent Neural Network (HPT-BiGRNN) technique for weather forecasting. The HPT-BiGRNN technique aims to utilize the past weather data for training the BiGRNN model and achieve the effective forecasts with minimum time duration. The BiGRNN is an enhanced version of Gated Recurrent Unit (GRU) that follows the process of passing input via forward and backward neural network and the outputs are linked to the identical output layer. The BiGRNN technique includes several hyper-parameters and hence, the hyperparameter optimization process takes place using Bird Mating Optimizer (BMO). The design of BMO algorithm for hyperparameter optimization of the BiGRNN, particularly for weather forecast shows the novelty of the work. The BMO algorithm is used to set hyperparameters such as momentum, learning rate, batch size and weight decay. The experimental result the HPT-BiGRNN approach has resulted in a lower RMSE of 0.173 whereas the Fuzzy-GP, Fuzzy-SC, MLP-ANN and RBF-ANN methods have gained an increased RMSE of 0.218, 0.216, 0.202 and 0.245 respectively.

Accurate experimental band gap predictions with multifidelity correction learning

To improve the precision of machine-learning predictions, we investigate various techniques that combine multiple quality sources for the same property. In particular, focusing on the electronic band gap, we aim at having the lowest error by taking advantage of all available experimental measurements and density-functional theory calculations. We show that learning about the difference between high- and low-quality values, considered a correction, significantly improves the results compared to learning on the sole high-quality experimental data. As a preliminary step, we also introduce an extension of the MODNet model, which consists of using a genetic algorithm for hyperparameter optimization. Thanks to this, MODNet is shown to achieve excellent performance on the Matbench test suite.

Post-Traumatic Stress Disorder (PTSD) biomarker identification using a deep learning model

Post-traumatic stress disorder (PTSD) is a neuropsychological disorder in which individuals struggle to recover from a traumatizing event. It affects a significant population, including COVID-19 patients, frontline health workers, and war veterans. Given biases associated with self-assessment and diagnosis of PTSD, researchers are actively searching for unbiased biological markers (biomarkers) for predicting PTSD status. The Systems Biology of PTSD Consortium has collected candidate biomarkers for PTSD using molecular and clinical measurements of male war veterans between the ages of 20 and 60. PTSD-positive and negative subjects were separated based on the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) Clinician Administered PTSD Scale (CAPS) scores, derived from structured interviews to measure an individual’s abundance of symptoms, such as re-experiencing, flashbacks, and hyperarousal. CAPS scores higher than 40 were considered PTSD-positive and below 20 were considered negative. We created artificial neural network models to classify PTSD-positive and negative individuals based on metabolomics, micro-RNA (miRNA), protein expression, endocrine markers, and DNA methylation datasets. Model training involved 64 iterations of a Bayesian Hyperparameter Optimization algorithm with 5-fold cross-validation. Each model was calibrated based on cross-validation performance and variance across iterations and then fit to the entire respective dataset (76 PTSD-positive, 76 PTSD-negative). We applied the trained models to an independent validation cohort to assess accuracy on unseen datasets. The top performing datasets from the validation cohort based on classification accuracy were metabolomics (65.2%) and protein expression (61.8%). We anticipate the candidate biomarkers identified in this and future studies will assist with the diagnosis of PTSD.

Ensemble of Deep Learning-Based Multimodal Remote Sensing Image Classification Model on Unmanned Aerial Vehicle Networks

Recently, unmanned aerial vehicles (UAVs) have been used in several applications of environmental modeling and land use inventories. At the same time, the computer vision-based remote sensing image classification models are needed to monitor the modifications over time such as vegetation, inland water, bare soil or human infrastructure regardless of spectral, spatial, temporal, and radiometric resolutions. In this aspect, this paper proposes an ensemble of DL-based multimodal land cover classification (EDL-MMLCC) models using remote sensing images. The EDL-MMLCC technique aims to classify remote sensing images into the different cloud, shades, and land cover classes. Primarily, median filtering-based preprocessing and data augmentation techniques take place. In addition, an ensemble of DL models, namely VGG-19, Capsule Network (CapsNet), and MobileNet, is used for feature extraction. In addition, the training process of the DL models can be enhanced by the use of hosted cuckoo optimization (HCO) algorithm. Finally, the salp swarm algorithm (SSA) with regularized extreme learning machine (RELM) classifier is applied for land cover classification. The design of the HCO algorithm for hyperparameter optimization and SSA for parameter tuning of the RELM model helps to increase the classification outcome to a maximum level considerably. The proposed EDL-MMLCC technique is tested using an Amazon dataset from the Kaggle repository. The experimental results pointed out the promising performance of the EDL-MMLCC technique over the recent state of art approaches.

Comparison of Tree-Structured Parzen Estimator Optimization in Three Typical Neural Network Models for Landslide Susceptibility Assessment

Landslides pose a constant threat to the lives and property of mountain people and may also cause geomorphological destruction such as soil and water loss, vegetation destruction, and land cover change. Landslide susceptibility assessment (LSA) is a key component of landslide risk evaluation. There are many related studies, but few analyses and comparisons of models for optimization. This paper aims to introduce the Tree-structured Parzen Estimator (TPE) algorithm for hyperparameter optimization of three typical neural network models for LSA in Shuicheng County, China, as an example, and to compare the differences of predictive ability among the models in order to achieve higher application performance. First, 17 influencing factors of landslide multiple data sources were selected for spatial prediction, hybrid ensemble oversampling and undersampling techniques were used to address the imbalanced sample and small sample size problem, and the samples were randomly divided into a training set and validation set. Second, deep neural network (DNN), recurrent neural network (RNN), and convolutional neural network (CNN) models were adopted to predict the regional landslides susceptibility, and the TPE algorithm was used to optimize the hyperparameters respectively to improve the assessment capacity. Finally, to compare the differences and optimization effects of these models, several objective measures were applied for validation. The results show that the high-susceptibility regions mostly distributed in bands along fault zones, where the lithology is mostly claystone, sandstone, and basalt. The DNN, RNN, and CNN models all perform well in LSA, especially the RNN model. The TPE optimization significantly improves the accuracy of the DNN and CNN (3.92% and 1.52%, respectively), but does not improve the performance of the RNN. In summary, our proposed RNN model and TPE-optimized DNN and CNN model have robust predictive capability for landslide susceptibility in the study area and can also be applied to other areas containing similar geological conditions.

Hybrid XGboost model with various Bayesian hyperparameter optimization algorithms for flood hazard susceptibility modeling

The purpose of this investigation is to develop an optimal model to flood susceptibility mapping in the Kan watershed, Tehran, Iran. Therefore, in this study, three Bayesian optimization hyper-parameter algorithms including Upper confidence bound (UCB), Probability of improvement (PI) and Expected improvement (EI) in order to Extreme Gradient Boosting (XGB) machine learning model optimization and Extreme randomize tree (ERT) model for modeling flood hazard were used. In order to perform flood susceptibility mapping, 118 historic flood locations were identified and analyzed using 17 geo-environmental explanatory variables to predict flooding susceptibility. Flood locations data were divided into 70% for training and 30% for testing of models developed. The receiver operating characteristic (ROC) curve parameters were used to evaluate the performance of the models. The evaluation results based on the criterion area under curve (AUC) in the testing stage showed that the ERT and XGB models have efficiencies of 91.37% and 91.95%, respectively. The evaluation of the efficiency of Bayesian hyperparameters optimization methods on the XGB model also showed that these methods increase the efficiency of the XGB model, so that the model efficiency using these methods EI-XGB, POI-XGB and UCB-XGB based on the AUC in the testing stage were 95.89%, 96.87% and 96.38%, respectively. The results of the relative importance of the five models shows that the variables of elevation and distance from the river are the significant compared to other variables in predicting flood hazard in the Kan watershed.

RHOASo: An Early Stop Hyper-Parameter Optimization Algorithm

This work proposes a new algorithm for optimizing hyper-parameters of a machine learning algorithm, RHOASo, based on conditional optimization of concave asymptotic functions. A comparative analysis of the algorithm is presented, giving particular emphasis to two important properties: the capability of the algorithm to work efficiently with a small part of a dataset and to finish the tuning process automatically, that is, without making explicit, by the user, the number of iterations that the algorithm must perform. Statistical analyses over 16 public benchmark datasets comparing the performance of seven hyper-parameter optimization algorithms with RHOASo were carried out. The efficiency of RHOASo presents the positive statistically significant differences concerning the other hyper-parameter optimization algorithms considered in the experiments. Furthermore, it is shown that, on average, the algorithm needs around 70% of the iterations needed by other algorithms to achieve competitive performance. The results show that the algorithm presents significant stability regarding the size of the used dataset partition.

A Novel Hybrid Deep Learning Model for Sugar Price Forecasting Based on Time Series Decomposition

Sugar price forecasting has attracted extensive attention from policymakers due to its significant impact on people’s daily lives and markets. In this paper, we present a novel hybrid deep learning model that utilizes the merit of a time series decomposition technology empirical mode decomposition (EMD) and a hyperparameter optimization algorithm Tree of Parzen Estimators (TPEs) for sugar price forecasting. The effectiveness of the proposed model was implemented in a case study with the price of London Sugar Futures. Two experiments are conducted to verify the superiority of the EMD and TPE. Moreover, the specific effects of EMD and TPE are analyzed by the DM test and improvement percentage. Finally, empirical results demonstrate that the proposed hybrid model outperforms other models.

A hybrid method for short-term wind speed forecasting based on Bayesian optimization and error correction

Accurate wind speed forecast plays an important role in the safe and stable operation of large-scale wind power integrated grid system. In this paper, a new hybrid model for short-term wind speed forecasting based on hyperparameter optimization and error correction is proposed, where the forecasting period is 5, 10, and 15 min, respectively, for three sites. The empirical wavelet transform is used to decompose the original wind speed series. Then, the Elman neural network and kernel extreme learning machine, which adopt Bayesian optimization algorithm for hyperparameter optimization, are used as predictors for wind speed prediction and error processing, respectively. In addition, a new error correction model using wind speed as model input is proposed. In order to verify the performance of the proposed model, three datasets collected from different real-world wind farms in Gansu and Xinjiang were considered as a case study to comprehensively evaluate the prediction performance of ten forecasting models. The results reveal that the proposed model has higher prediction accuracy and better prediction performance than the contrast models.

A scalable algorithm for the optimization of neural network architectures

We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

Optimization of SMOTE for imbalanced data based on AdaRBFNN and hybrid metaheuristics

Oversampling ratio N and the minority class’ nearest neighboring number k are key hyperparameters of synthetic minority oversampling technique (SMOTE) to reconstruct the class distribution of dataset. No optimal default value exists there. Therefore, it is of necessity to discuss the influence of the output dataset on the classification performance when SMOTE adopts various hyperparameter combinations. In this paper, we propose a hyperparameter optimization algorithm for imbalanced data. By iterating to find reasonable N and k for SMOTE, so as to build a balanced and high-quality dataset. As a result, a model with outstanding performance and strong generalization ability is trained, thus effectively solving imbalanced classification. The proposed algorithm is based on the hybridization of simulated annealing mechanism (SA) and particle swarm optimization algorithm (PSO). In the optimization, Cohen’s Kappa is used to construct the fitness function, and AdaRBFNN, a new classifier, is integrated by multiple trained RBF neural networks based on AdaBoost algorithm. Kappa of each generation is calculated according to the classification results, so as to evaluate the quality of candidate solution. Experiments are conducted on seven groups of KEEL datasets. Results show that the proposed algorithm delivers excellent performance and can significantly improve the classification accuracy of the minority class.

Dynamical Hyperparameter Optimization via Deep Reinforcement Learning in Tracking.

Hyperparameters are numerical pre-sets whose values are assigned prior to the commencement of a learning process. Selecting appropriate hyperparameters is often critical for achieving satisfactory performance in many vision problems, such as deep learning-based visual object tracking. However, it is often difficult to determine their optimal values, especially if they are specific to each video input. Most hyperparameter optimization algorithms tend to search a generic range and are imposed blindly on all sequences. In this paper, we propose a novel dynamical hyperparameter optimization method that adaptively optimizes hyperparameters for a given sequence using an action-prediction network leveraged on continuous deep Q-learning. Since the observation space for object tracking is significantly more complex than those in traditional control problems, existing continuous deep Q-learning algorithms cannot be directly applied. To overcome this challenge, we introduce an efficient heuristic strategy to handle high dimensional state space, while also accelerating the convergence behavior. The proposed algorithm is applied to improve two representative trackers, a Siamese-based one and a correlation-filter-based one, to evaluate its generalizability. Their superior performances on several popular benchmarks are clearly demonstrated. Our source code is available at https://github.com/shenjianbing/dqltracking.

An Effective Hyperparameter Optimization Algorithm for DNN to Predict Passengers at a Metro Station

As one of the public transportation systems, metro is certainly an indispensable part in urban areas of a metropolis today. Several successful results have shown that deep learning technologies might provide an effective way to predict the number of passengers at a metro station. However, most information systems based on deep learning technologies are usually designed and tuned manually by using domain knowledge and trial-and-error; thus, how to find out a set of suitable hyperparameters for a deep neural network (DNN) has become a critical research issue. To deal with the problem of hyperparameter setting for a DNN in solving the prediction of passengers at a metro station, a novel metaheuristic algorithm called search economics for hyperparameter optimization is presented to improve the accuracy rate of such a prediction system. The basic idea of the proposed algorithm is to divide the solution space into a set of subspaces first and then assign a different number of search agents to each subspace based on the “potential of each subspace.” The potential is estimated based on the objective values of the searched solutions, the objective values of the probe solutions, and the computation time invested in each subspace. The proposed method is compared with Bayesian, random forest, support vector regression, DNN, and DNN with different hyperparameter search algorithms, namely, grid search, simulated annealing, and particle swarm optimization. The simulation results using the data provided by the government of Taipei city, Taiwan, indicate that the proposed method outperforms all the other forecasting methods compared in this article in terms of the mean absolute percentage error.

Evolutionary Hybrid System for Energy Consumption Forecasting for Smart Meters

The usage of smart grids is growing steadily around the world. This technology has been proposed as a promising solution to enhance energy efficiency and improve consumption management in buildings. Such benefits are usually associated with the ability of accurately forecasting energy demand. However, the energy consumption series forecasting is a challenge for statistical linear and Machine Learning (ML) techniques due to temporal fluctuations and the presence of linear and non-linear patterns. Traditional statistical techniques are able to model linear patterns, while obtaining poor results in forecasting the non-linear component of the time series. ML techniques are data-driven and can model non-linear patterns, but their feature selection process and parameter specification are a complex task. This paper proposes an Evolutionary Hybrid System (EvoHyS) which combines statistical and ML techniques through error series modeling. EvoHyS is composed of three phases: (i) forecast of the linear and seasonal component of the time series using a Seasonal Autoregressive Integrated Moving Average (SARIMA) model, (ii) forecast of the error series using an ML technique, and (iii) combination of both linear and non-linear forecasts from (i) and (ii) using a a secondary ML model. EvoHyS employs a Genetic Algorithm (GA) for feature selection and hyperparameter optimization in phases (ii) and (iii) aiming to improve its accuracy. An experimental evaluation was conducted using consumption energy data of a smart grid in a one-step-ahead scenario. The proposed hybrid system reaches statistically significant improvements when compared to other statistical, hybrid, and ML approaches from the literature utilizing well known metrics, such as Mean Squared Error (MSE).