Simple Deterministic Selection-Based Genetic Algorithm for Hyperparameter Tuning of Machine Learning Models

Ismail Damilola Raji,Ime Jarlath Umoh,Habeeb Bello-Salau,Mutiu Adesina Adegboye,Adeiza James Onumanyi,Ahmed Tijani Salawudeen

doi:10.3390/app12031186

Ismail Damilola Raji, Ime Jarlath Umoh + Show 4 more

Open Access

https://doi.org/10.3390/app12031186

Copy DOI

Abstract

Hyperparameter tuning is a critical function necessary for the effective deployment of most machine learning (ML) algorithms. It is used to find the optimal hyperparameter settings of an ML algorithm in order to improve its overall output performance. To this effect, several optimization strategies have been studied for fine-tuning the hyperparameters of many ML algorithms, especially in the absence of model-specific information. However, because most ML training procedures need a significant amount of computational time and memory, it is frequently necessary to build an optimization technique that converges within a small number of fitness evaluations. As a result, a simple deterministic selection genetic algorithm (SDSGA) is proposed in this article. The SDSGA was realized by ensuring that both chromosomes and their accompanying fitness values in the original genetic algorithm are selected in an elitist-like way. We assessed the SDSGA over a variety of mathematical test functions. It was then used to optimize the hyperparameters of two well-known machine learning models, namely, the convolutional neural network (CNN) and the random forest (RF) algorithm, with application on the MNIST and UCI classification datasets. The SDSGA’s efficiency was compared to that of the Bayesian Optimization (BO) and three other popular metaheuristic optimization algorithms (MOAs), namely, the genetic algorithm (GA), particle swarm optimization (PSO) and biogeography-based optimization (BBO) algorithms. The results obtained reveal that the SDSGA performed better than the other MOAs in solving 11 of the 17 known benchmark functions considered in our study. While optimizing the hyperparameters of the two ML models, it performed marginally better in terms of accuracy than the other methods while taking less time to compute.

Highlights

Machine learning (ML) techniques are increasingly being used in a wide range of research areas including for object detection and classification [1,2], natural language processing and data-driven control [3]
The performance of the simple deterministic selection genetic algorithm (SDSGA) is compared to other metaheuristic algorithms both on the different benchmark functions and for fine-tuning the hyperparameters of two machine learning (ML) models
This section discusses how the proposed SDSGA performs when different percentages P of the total population size were chosen as the number of parents, ranging from 10% to 90% with a 10% increment, when applied to the benchmark test functions, as well as how the considered ML models, namely the random forest and the convolutional neural network (CNN) performed when trained on the MNIST and UCI datasets

Summary

Introduction

Machine learning (ML) techniques are increasingly being used in a wide range of research areas including for object detection and classification [1,2], natural language processing and data-driven control [3]. Despite the relatively good performance obtained, selecting the optimal hyperparameter (HP) values for most ML models remains a significant challenge. These hyperparameters are usually classified as either continuous, discrete, or a combination of the two [4]. Continuous hyperparameter metrics that can be optimized include the learning rate, decay, momentum and regularization parameters [5,6]. We could perhaps note that other conditional search spaces might be influenced by other hyperparameters, such as the type of optimizer used in training, which determines the types and number of hyperparameters to be used [8]

Methods

Results

Conclusion