An Optimization of Neural Network Hyper-Parameter to Increase Its Performance

Yinxuan Fu

doi:10.4236/iim.2018.104008

Abstract

With the boost of artificial intelligence, the study of neural network intrigues scientists. Artificial neural network, which was first designed theoretically in 1943 based on understanding of human brains, demonstrated impressing computational and learning capabilities. In this paper, we investigated the neural network’s learning capability by using a feed-forward neural network to recognize human’s digit hand-writing. Controlled experiments were executed by changing the input values of different parameters, such as learning rates and hidden layer units. After investigating upon the effects of each parameter on the overall learning performance of the neural network, we concluded that, when an intermediate value of one given parameter was implemented, the neural network achieved the highest learning efficiency, and potential problems like over-fitting would be prevented.

Highlights

The human brain has always intrigued scientists
The computational structure became the basis on which future development of artificial neural network was built
The basic structure of neural network consists of large number of artificial neurons, which execute similar function as biological neurons, but in more abstract forms [3]

Summary

Introduction

The human brain has always intrigued scientists. The brain’s function is very powerful and efficient [1]. The basic structure of neural network consists of large number of artificial neurons, which execute similar function as biological neurons, but in more abstract forms [3]. Like the human brain, neural network will “behave” differently if input data is different or if different network parameters are applied. Such parameters include learning rate (the rate at which a neural network is adapting to new information and/or data), weights (the measure of influence of one neuron to another), number of hidden layers, etc. We don’t implement any image pre-processing since it is beyond the scope of this paper; the input layer contains 256 neurons, one for each pixel.

Optimization

Generalization

Initialization

Conclusion