A BAYESIAN APPROACH BASED ON ACQUISITION FUNCTION FOR OPTIMAL SELECTION OF DEEP LEARNING HYPERPARAMETERS: A CASE STUDY WITH ENERGY MANAGEMENT DATA

Muhammad Ali,Krishneel Prakash Krishneel Prakash,Hemanshu Pota Hemanshu Pota

doi:10.31580/sps.v2i1.1232

Muhammad Ali, Krishneel Prakash Krishneel Prakash + Show 1 more

Open Access

https://doi.org/10.31580/sps.v2i1.1232

Copy DOI

Abstract

With the recent rollout of smart meters, huge amount of data can be generated on hourly and daily basis. Researchers and industry persons can leverage from this big data to make intelligent decisions via deep learning (DL) algorithms. However, the performance of DL algorithms are heavily dependent on the proper selection of parameters. If the hyperparameters are poorly selected, they usually lead to suboptimal results. Traditional approaches include a manual setting of parameters by trial and error methods which is time consuming and difficult process. In this paper, a Bayesian approach based on acquisition is presented to automatic selection of optimal parameters based on provided data. The acquisition function was established to search for the best parameter from the input space and evaluate the next points based on past observations. The tuning process identifies the best model parameters by iterating the objective function and minimizing the loss for optimizable variables such as learning rate and Hidden layersize. To validate the presented approach, we conducted a case study on real-life energy management datasets while constructing a deep learning model on MATLAB platform. A performance comparison was drawn with random parameters and optimal parameters selected by presented approach. The comparison results illustrate that the presented approach is effective as it brings a notable improvement in the performance of learning algorithm.

Highlights

One of the key challenges in the implementation of deep learning (DL) algorithms is the correct settings of its parameters to achieve optimal results
The need for tuning DL parameters is ubiquitous in engineering industry as they heavily influence the performance of learning algorithm
The methods like random search [4], grid search [5], evolutionary search [6] and guided search [7] are used heavily in the past. These methods are relatively inefficient for the automatic selection of parameters as they do not choose the hyperparameters based on previous results

Summary

Graphical Results

Fig: 1(a): Acquisition function plot with optimizable parameters such as hidden layer size and learning rate. Fig: 1(b): Objective function plot to determines the best feasible parameters. The methods like random search [4], grid search [5], evolutionary search [6] and guided search [7] are used heavily in the past These methods are relatively inefficient for the automatic selection of parameters as they do not choose the hyperparameters based on previous results. These methods need a significant amount of time for calculation of parameters which is not desirable in practical applications. A practical approach based on Bayesian optimization and acquisition function was established for the automatic selection of DL parameters. The study aimed at identifying the best feasible parameters while conducting the experiments with real-life energy management datasets

Methodology

Results

Findings