Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics

Laurits Tani,Christian Veelken,Mario Kadastik,Diana Rand

doi:10.1140/epjc/s10052-021-08950-y

Abstract

The analysis of vast amounts of data constitutes a major challenge in modern high energy physics experiments. Machine learning (ML) methods, typically trained on simulated data, are often employed to facilitate this task. Several choices need to be made by the user when training the ML algorithm. In addition to deciding which ML algorithm to use and choosing suitable observables as inputs, users typically need to choose among a plethora of algorithm-specific parameters. We refer to parameters that need to be chosen by the user as hyperparameters. These are to be distinguished from parameters that the ML algorithm learns autonomously during the training, without intervention by the user. The choice of hyperparameters is conventionally done manually by the user and often has a significant impact on the performance of the ML algorithm. In this paper, we explore two evolutionary algorithms: particle swarm optimization and genetic algorithm, for the purposes of performing the choice of optimal hyperparameter values in an autonomous manner. Both of these algorithms will be tested on different datasets and compared to alternative methods.

Highlights

The subject of this paper is to describe two evolutionary algorithms [6], which allow to find a set of optimal hyperparameters in an autonomous manner
One considers a mapping from a point h in hyperparameter space H to a “score” value s(h), which quantifies the performance of the Machine learning (ML) algorithm for a given task
The performance of both evolutionary algorithms, particle swarm optimization (PSO) and genetic algorithm (GA), is evaluated on two tasks: on the Rosenbrock function, which provides an example for a difficult function minimization problem, and on the ATLAS Higgs boson machine learning (ML) challenge, as a typical application of ML methods in high energy physics (HEP)

Summary

Introduction

The subject of this paper is to describe two evolutionary algorithms [6], which allow to find a set of optimal hyperparameters in an autonomous manner. The evolutionary algorithms studied in this paper are particle swarm optimization (PSO) [7] and genetic algorithm (GA) [8]. The task of finding optimal hyperparameter values can be recast as function maximization. One considers a mapping from a point h in hyperparameter space H to a “score” value s(h), which quantifies the performance of the ML algorithm for a given task. Using a suitable encoding for hyperparameters of non-floating-point type, the hyperparameter space H can be taken to be the Euclidean space IRN, with N denoting the number of hyperparameters. The optimal hyperparameters, denoted by the symbol h, are those that satisfy the condition:

Methods

Discussion

Conclusion