Algorithm Hyperparameter Optimization Research Articles

Convolutional Neural Networks-Based Lung Nodule Classification: A Surrogate-Assisted Evolutionary Algorithm for Hyperparameter Optimization

This article investigates deep neural networks (DNNs)-based lung nodule classification with hyperparameter optimization. Hyperparameter optimization in DNNs is a computationally expensive problem, and a surrogate-assisted evolutionary algorithm has been recently introduced to automatically search for optimal hyperparameter configurations of DNNs, by applying computationally efficient surrogate models to approximate the validation error function of hyperparameter configurations. Different from existing surrogate models adopting stationary covariance functions (kernels) to measure the difference between hyperparameter points, this article proposes a nonstationary kernel that allows the surrogate model to adapt to functions whose smoothness varies with the spatial location of inputs. A multilevel convolutional neural network (ML-CNN) is built for lung nodule classification, and the hyperparameter configuration is optimized by the proposed nonstationary kernel-based Gaussian surrogate model. Our algorithm searches with a surrogate for optimal setting via a hyperparameter importance-based evolutionary strategy, and the experiments demonstrate our algorithm outperforms manual tuning and several well-established hyperparameter optimization methods, including random search, grid search, the tree-structured parzen estimator (TPE) approach, Gaussian processes (GP) with stationary kernels, and the recently proposed hyperparameter optimization via RBF and dynamic (HORD) coordinate search.

A Comparative Study of Machine Learning Models with Hyperparameter Optimization Algorithm for Mapping Mineral Prospectivity

Selecting internal hyperparameters, which can be set by the automatic search algorithm, is important to improve the generalization performance of machine learning models. In this study, the geological, remote sensing and geochemical data of the Lalingzaohuo area in Qinghai province were researched. A multi-source metallogenic information spatial data set was constructed by calculating the Youden index for selecting potential evidence layers. The model for mapping mineral prospectivity of the study area was established by combining two swarm intelligence optimization algorithms, namely the bat algorithm (BA) and the firefly algorithm (FA), with different machine learning models. The receiver operating characteristic (ROC) and prediction-area (P-A) curves were used for performance evaluation and showed that the two algorithms had an obvious optimization effect. The BA and FA differentiated in improving multilayer perceptron (MLP), AdaBoost and one-class support vector machine (OCSVM) models; thus, there was no optimization algorithm that was consistently superior to the other. However, the accuracy of the machine learning models was significantly enhanced after optimizing the hyperparameters. The area under curve (AUC) values of the ROC curve of the optimized machine learning models were all higher than 0.8, indicating that the hyperparameter optimization calculation was effective. In terms of individual model improvement, the accuracy of the FA-AdaBoost model was improved the most significantly, with the AUC value increasing from 0.8173 to 0.9597 and the prediction/area (P/A) value increasing from 3.156 to 10.765, where the mineral targets predicted by the model occupied 8.63% of the study area and contained 92.86% of the known mineral deposits. The targets predicted by the improved machine learning models are consistent with the metallogenic geological characteristics, indicating that the swarm intelligence optimization algorithm combined with the machine learning model is an efficient method for mineral prospectivity mapping.

Design of Optimal Deep Learning Based Human Activity Recognition on Sensor Enabled Internet of Things Environment

In recent times, Human activity recognition (HAR) becomes a major challenging issue among computer vision applications in day to day lives. The HAR is mainly envisioned to be utilized with other technologies, namely Internet of Things (IoT) and sensor technologies. Due to the advancements of deep learning (DL) approaches, automated high level feature extraction process can be utilized to improve the HAR results. In addition, DL techniques can be employed in different domains of sensor enabled HAR. In this aspect, this study designs an optimal DL based HAR (ODL-HAR) model on sensor enabled IoT environments. The ODL-HAR technique aims to determine human activities in day to day lives using wearables and IoT devices. The proposed ODL-HAR technique involves different stages of operations namely data acquisition, data preprocessing, feature extraction, classification, and parameter optimization. The ODL-HAR technique uses a MobileNet-v2 model as a feature extractor and bidirectional long short-term memory (BiLSTM) model as a classifier. In order to optimally tune the hyperparameter involved in the BiLSTM model, chaos game optimization (CGO) algorithm is employed and thereby raises the recognition performance. The design of CGO algorithm for hyperparameter optimization of HAR shows the novelty of the work. A wide range of simulations takes place to point out the betterment of the ODL-HAR technique on two benchmark datasets. The experimental results portrayed the enhanced performance of the ODL-HAR technique over the other recent HAR approaches interms of different evaluation parameters.

Fast Circulant Tensor Power Method for High-Order Principal Component Analysis

To understand high-order intrinsic key patterns in high-dimensional data, tensor decomposition is a more versatile tool for data analysis than standard flat-view matrix models. Several existing tensor models aim to achieve rapid computation of high-order principal components based on the tensor power method. However, since a tensor power method does not enforce orthogonality in subsequently calculated decomposition components, it causes far more challenges on principal component analysis of high-order tensors. To address this problem, several tensor power method variant algorithms incorporating sparsity into decomposition factors have been proposed. However, because these variant algorithms require additional procedures based on data-driven hyper-parameter optimization algorithms, a trade-off between computational cost and convergence exists. In this paper, a novel tensor power method called the fast circulant tensor power method is proposed. The proposed algorithm combines tensor-train decomposition and the power method. Tensor-train decomposition is a high-order tensor decomposition method based on auxiliary unfolding matrix decomposition. Thus, the power method can be embedded into our methodology without any additional processes. Notably, a simple combination of these two methods may cause a local optima problem because the power method only guarantees convergence on each unfolding matrix in tensor-train decomposition. To solve this problem, the circulant updating method is proposed, which globally optimizes all factor vectors by reordering some steps of the factor vector updates. It is experimentally demonstrated that, compared to state-of-the-art tensor power method variant methodologies, the proposed algorithm achieves the lowest computational complexity and quantitatively good performance in various applications including large-scale color image decomposition and convolutional neural network compression.

ExperienceThinking: Constrained hyperparameter optimization based on knowledge and pruning

Machine learning models are very sensitive to the hyperparameters, and their evaluations are generally expensive. Users desperately need intelligent methods to quickly optimize hyperparameter settings according to known evaluation information, so as to effectively promote the performance of the machine learning models within the limited and small budget. Motivated by this, in this paper, we propose ExperienceThinking algorithm to quickly find the best possible hyperparameter configuration of machine learning algorithms within a few configuration evaluations. ExperienceThinking designs two novel approaches, which make full use of the known evaluation information to intelligently infer optimal configurations from two aspects: search space pruning and knowledge utilization respectively. Two approaches suit for two different kinds of constrained hyperparameter optimization problems, they complement with each other and their combination increases the generality and effectiveness of the ExperienceThinking. To demonstrate the benefit of ExperienceThinking, we conduct extensive experiments using various constrained hyperparameter optimization problems, and compare it with classic hyperparameter optimization algorithms. The experimental results present that our proposed algorithm provides superior results and the design of our proposed algorithm is reasonable.

Detecting Electricity Theft Cyber-Attacks in AMI Networks Using Deep Vector Embeddings

Despite being equipped with advanced metering infrastructure (AMI), utility companies are subjected to electricity theft cyber-attacks. The existing machine learning-based detectors do not capture well the complex patterns and the temporal correlation present in the time-series profile of energy consumption data. This article proposes a deep recurrent vector embedding model to identify electricity theft cyber-attacks. Vector embedding is a data representation method that we use to express energy consumption profiles as vectors of real numbers. Since the reported electricity readings may be benign or malicious, vector embedding algorithms help in analyzing the relationships and capturing the patterns within the customer's reported readings. Furthermore, our model captures well the time-series nature of the data due to the adoption of gated recurrent units. We implement a sequential grid-search hyperparameter optimization algorithm to further improve the models detection performance. We test our model against two real datasets of benign and malicious readings. Results are 95.8% in detection rate (DR), 2.1% in false alarm (FA), and 93.7% in highest difference (HD). Our model outperforms shallow detectors by 3.5%-9.7% in DR, 3.1%-10% in FA, and 8.7%-21.8% in HD. It also outperforms deep detectors by 1.5%-3.2% in DR, 2%-4.3% in FA, and 5.6%-9.6% in HD.

An Improved AdaBoost Algorithm for Hyperparameter Optimization

AdaBoost algorithm is a typical Boosting algorithm, which belongs to a successful representative in the Boosting family. This algorithm can upgrade a weak classifier with a better classification effect than random classification to a strong classifier with high classification accuracy, where n_estimators represents the number of iterations of the base classifier. If the value is too large, it will easily cause the model to overfit, if it is too small, it is easy. The model is under-fitting, and the parameter setting is not set randomly, but according to the current status of the data set. Aiming at the problem that the number of iterations in the AdaBoost algorithm is uncertain, this paper introduces a Bayesian optimization algorithm for hyperparameter tuning, which makes the value of hyper parameter in AdaBoost algorithm suitable for the current data set, and finally obtains a hyperparameter optimization AdaBoost algorithm. The experiment result shows the method that adopt Bayesian optimization algorithm for hyperparameter optimization and apply the optimized hyperparameter value to the AdaBoost algorithm does not only improves the classification accuracy of the AdaBoost algorithm, but also avoids overfitting and underfitting of the model.

Automatic tuning of hyperparameters using Bayesian optimization

Deep learning is a field in artificial intelligence that works well in computer vision, natural language processing and audio recognition. Deep neural network architectures has number of layers to conceive the features well, by itself. The hyperparameter tuning plays a major role in every dataset which has major effect in the performance of the training model. Due to the large dimensionality of data it is impossible to tune the parameters by human expertise. In this paper, we have used the CIFAR-10 Dataset and applied the Bayesian hyperparameter optimization algorithm to enhance the performance of the model. Bayesian optimization can be used for any noisy black box function for hyperparameter tuning. In this work Bayesian optimization clearly obtains optimized values for all hyperparameters which saves time and improves performance. The results also show that the error has been reduced in graphical processing unit than in CPU by 6.2% in the validation. Achieving global optimization in the trained model helps transfer learning across domains as well.

Application of Improved LightGBM Model in Blood Glucose Prediction

In recent years, with increasing social pressure and irregular schedules, many people have developed unhealthy eating habits, which has resulted in an increasing number of patients with diabetes, a disease that cannot be cured under the current medical conditions, and can only be mitigated by early detection and prevention. A lot of human and material resources are required for the detection of the blood glucose of a large number of people in medical examination, while the integrated learning model based on machine learning can quickly predict the blood glucose level and assist doctors in treatment. Therefore, an improved LightGBM model based on the Bayesian hyper-parameter optimization algorithm is proposed for the prediction of blood glucose, namely HY_LightGBM, which optimizes parameters using a Bayesian hyper-parameter optimization algorithm based on LightGBM. The Bayesian hyper-parameter optimization algorithm is a model-based method for finding the minimum value of the function so as to obtain the optimal parameters of the LightGBM model. Experiments have demonstrated that the parameters obtained by the Bayesian hyper-parameter optimization algorithm are superior to those obtained by a genetic algorithm and random search. The improved LightGBM model based on the Bayesian hyper-parameter optimization algorithm achieves a mean square error of 0.5961 in blood glucose prediction, with a higher accuracy than the XGBoost model and CatBoost model.

Optimized neural networks in industrial data analysis

The research work presented here tackles the problem of finding the optimum value for hyper-parameters, such as number of layers and number of neurons per layer, for a fully-connected Artificial Neural Network (ANN), particularly in regression problems. A proposed optimization strategy is tested on different datasets related to diverse industrial applications: (1) prediction of the performance of exploration algorithms for mobile robots, (2) prediction of the compressive strength of concrete, (3) prediction of energy output from a power plant; and (4) prediction of wine quality. Different evaluation metrics, such as Pearson correlation coefficient (R), Square Correlation Coefficient (R2), Absolute Relative Error (RAE), Root Mean Square Error (RMSE), and Mean Absolute Error (MAE), are used to determine the best performing prediction model when the hyper-parameter optimization algorithms are used. That result is compared to the performance of the best model previously reported in the literature on the same dataset in order to determine the gains achieved by using the hyper-parameter optimization strategy under test.

Basic Enhancement Strategies When Using Bayesian Optimization for Hyperparameter Tuning of Deep Neural Networks

Compared to the traditional machine learning models, deep neural networks (DNN) are known to be highly sensitive to the choice of hyperparameters. While the required time and effort for manual tuning has been rapidly decreasing for the well developed and commonly used DNN architectures, undoubtedly DNN hyperparameter optimization will continue to be a major burden whenever a new DNN architecture needs to be designed, a new task needs to be solved, a new dataset needs to be addressed, or an existing DNN needs to be improved further. For hyperparameter optimization of general machine learning problems, numerous automated solutions have been developed where some of the most popular solutions are based on Bayesian Optimization (BO). In this work, we analyze four fundamental strategies for enhancing BO when it is used for DNN hyperparameter optimization. Specifically, diversification, early termination, parallelization, and cost function transformation are investigated. Based on the analysis, we provide a simple yet robust algorithm for DNN hyperparameter optimization - DEEP-BO (Diversified, Early-termination-Enabled, and Parallel Bayesian Optimization). When evaluated over six DNN benchmarks, DEEP-BO mostly outperformed well-known solutions including GP-Hedge, BOHB, and the speed-up variants that use Median Stopping Rule or Learning Curve Extrapolation. In fact, DEEP-BO consistently provided the top, or at least close to the top, performance over all the benchmark types that we have tested. This indicates that DEEP-BO is a robust solution compared to the existing solutions. The DEEP-BO code is publicly available at https://github.com/snu-adsl/DEEP-BO.

An Intelligent Approach to Credit Card Fraud Detection Using an Optimized Light Gradient Boosting Machine

New advances in electronic commerce systems and communication technologies have made the credit card the potentially most popular method of payment for both regular and online purchases; thus, there is significantly increased fraud associated with such transactions. Fraudulent credit card transactions cost firms and consumers large financial losses every year, and fraudsters continuously attempt to find new technologies and methods for committing fraudulent transactions. The detection of fraudulent transactions has become a significant factor affecting the greater utilization of electronic payment. Thus, there is a need for efficient and effective approaches for detecting fraud in credit card transactions. This paper proposes an intelligent approach for detecting fraud in credit card transactions using an optimized light gradient boosting machine (OLightGBM). In the proposed approach, a Bayesian-based hyperparameter optimization algorithm is intelligently integrated to tune the parameters of a light gradient boosting machine (LightGBM). To demonstrate the effectiveness of our proposed OLightGBM for detecting fraud in credit card transactions, experiments were performed using two real-world public credit card transaction data sets consisting of fraudulent transactions and legitimate ones. Based on a comparison with other approaches using the two data sets, the proposed approach outperformed the other approaches and achieved the highest performance in terms of accuracy (98.40%), Area under receiver operating characteristic curve (AUC) (92.88%), Precision (97.34%) and F1-score (56.95%).

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Hyperparameters are important for machine learning algorithms since they directly control the behaviors of training algorithms and have a significant effect on the performance of machine learning models. Several techniques have been developed and successfully applied for certain application domains. However, this work demands professional knowledge and expert experience. And sometimes it has to resort to the brute-force search. Therefore, if an efficient hyperparameter optimization algorithm can be developed to optimize any given machine learning method, it will greatly improve the efficiency of machine learning. In this paper, we consider building the relationship between the performance of the machine learning models and their hyperparameters by Gaussian processes. In this way, the hyperparameter tuning problem can be abstracted as an optimization problem and Bayesian optimization is used to solve the problem. Bayesian optimization is based on the Bayesian theorem. It sets a prior over the optimization function and gathers the information from the previous sample to update the posterior of the optimization function. A utility function selects the next sample point to maximize the optimization function. Several experiments were conducted on standard test datasets. Experiment results show that the proposed method can find the best hyperparameters for the widely used machine learning models, such as the random forest algorithm and the neural networks, even multi-grained cascade forest under the consideration of time cost.

Comprehensive analysis of gradient-based hyperparameter optimization algorithms

The paper investigates hyperparameter optimization problem. Hyperparameters are the parameters of model parameter distribution. The adequate choice of hyperparameter values prevents model overfit and allows it to obtain higher predictive performance. Neural network models with large amount of hyperparameters are analyzed. The hyperparameter optimization for models is computationally expensive. The paper proposes modifications of various gradient-based methods to simultaneously optimize many hyperparameters. The paper compares the experiment results with the random search. The main impact of the paper is hyperparameter optimization algorithms analysis for the models with high amount of parameters. To select precise and stable models the authors suggest to use two model selection criteria: cross-validation and evidence lower bound. The experiments show that the models optimized using the evidence lower bound give higher error rate than the models obtained using cross-validation. These models also show greater stability when data is noisy. The evidence lower bound usage is preferable when the model tends to overfit or when the cross-validation is computationally expensive. The algorithms are evaluated on regression and classification datasets.

The Tabu_Genetic Algorithm: A Novel Method for Hyper-Parameter Optimization of Learning Algorithms

Machine learning algorithms have been widely used to deal with a variety of practical problems such as computer vision and speech processing. But the performance of machine learning algorithms is primarily affected by their hyper-parameters, as without good hyper-parameter values the performance of these algorithms will be very poor. Unfortunately, for complex machine learning models like deep neural networks, it is very difficult to determine their hyper-parameters. Therefore, it is of great significance to develop an efficient algorithm for hyper-parameter automatic optimization. In this paper, a novel hyper-parameter optimization methodology is presented to combine the advantages of a Genetic Algorithm and Tabu Search to achieve the efficient search for hyper-parameters of learning algorithms. This method is defined as the Tabu_Genetic Algorithm. In order to verify the performance of the proposed algorithm, two sets of contrast experiments are conducted. The Tabu_Genetic Algorithm and other four methods are simultaneously used to search for good values of hyper-parameters of deep convolutional neural networks. Experimental results show that, compared to Random Search and Bayesian optimization methods, the proposed Tabu_Genetic Algorithm finds a better model in less time. Whether in a low-dimensional or high-dimensional space, the Tabu_Genetic Algorithm has better search capabilities as an effective method for finding the hyper-parameters of learning algorithms. The presented method in this paper provides a new solution for solving the hyper-parameters optimization problem of complex machine learning models, which will provide machine learning algorithms with better performance when solving practical problems.

Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks

Most learning algorithms require the practitioner to manually set the values of many hyperparameters before the learning process can begin. However, with modern algorithms, the evaluation of a given hyperparameter setting can take a considerable amount of time and the search space is often very high-dimensional. We suggest using a lower-dimensional representation of the original data to quickly identify promising areas in the hyperparameter space. This information can then be used to initialize the optimization algorithm for the original, higher-dimensional data. We compare this approach with the standard procedure of optimizing the hyperparameters only on the original input. We perform experiments with various state-of-the-art hyperparameter optimization algorithms such as random search, the tree of parzen estimators (TPEs), sequential model-based algorithm configuration (SMAC), and a genetic algorithm (GA). Our experiments indicate that it is possible to speed up the optimization process by using lower-dimensional data representations at the beginning, while increasing the dimensionality of the input later in the optimization process. This is independent of the underlying optimization procedure, making the approach promising for many existing hyperparameter optimization algorithms.

Genetic algorithms for hyperparameter optimization in predictive business process monitoring

Predictive business process monitoring aims at predicting the outcome of ongoing cases of a business process based on past execution traces. A wide range of techniques for this predictive task have been proposed in the literature. It turns out that no single technique, under a default configuration, consistently achieves the best predictive accuracy across all datasets. Thus, the selection and configuration of a technique needs to be done for each dataset. This paper presents a framework for predictive process monitoring that brings together a range of techniques, each with an associated set of hyperparameters. The framework incorporates two automatic hyperparameter optimization algorithms, which, given a dataset, select suitable techniques for each step in the framework and configure these techniques with minimal user input. The proposed framework and hyperparameter optimization algorithms have been evaluated on two real-life datasets and compared with state-of-the-art approaches for predictive business process monitoring. The results demonstrate the scalability of the approach and its ability to identify accurate and reliable framework configurations.

Reinforcement-based Method for Simultaneous Clustering Algorithm Selection and its Hyperparameters Optimization

A wide range of clustering algorithms exists, most of them expose many hyperparameters, on which clustering partition quality depends. Simultaneous algorithm (model) selection and its hyperparameters optimization is considered to be a sophisticated task, which is known according to some sources as combined algorithm selection and hyperparameter optimization. In this paper, we focus on problem of selecting a clustering algorithm and its hyperparameter vector simultaneously given a dataset in order to achieve the best partition quality. We propose a method for selecting a proper clustering algorithm and its hyperparameter vector using reinforcement learning. Instead of tuning hyperparameters for all available clustering algorithms and selecting one showing the best performance, we make them to compete for time that they can use for optimizing their own hyperparameters. In our algorithm, we use a framework for multi-armed bandit problem, which is a special case of reinforcement learning. Each clustering algorithm is considered as an arm in the multi-armed bandit setting, while assigning a time budget to optimize hyperparameters of a clustering algorithm is considered as playing the corresponding arm. We conducted series of experiments for comparing out reinforcement learning approach to the classical exhaustive search approach. We conducted experiments on 20 datasets from UCI Repository such as Iris, haberman, krvskp, glass and other. We use 19 cluster validity indices to validate the clusters, built by selected and configured algorithm. As a hyperparameter optimization algorithm, we used SMAC. Our approach managed to improve model selection and hyperparameter optimization process, by sustaining the exploration-exploitation trade-off and spending available time budget more wisely.

An effective algorithm for hyperparameter optimization of neural networks

A major challenge in designing neural network (NN) systems is to determine the best structure and parameters for the network given the data for the machine learning problem at hand. Examples of parameters are the number of layers and nodes, the learning rates, and the dropout rates. Typically, these parameters are chosen based on heuristic rules and are manually fine-tuned, which may be very time-consuming, because evaluating the performance of a single parametrization of the NN may require several hours. In this paper, we address the problem of choosing appropriate parameters for the NN by formulating it as a box-constrained mathematical optimization problem and applying a derivative-free optimization tool that automatically and effectively searches the parameter space. The optimization tool employs a radial basis function model of the objective function (the prediction accuracy of the NN) to accelerate the discovery of configurations yielding high accuracy. Candidate configurations explored by the algorithm are trained to a small number of epochs, and only the most promising candidates receive full training. The performance of the proposed methodology is assessed on benchmark sets and in the context of predicting drug–drug interactions, showing promising results. The optimization tool used in this paper is open source.