Investigation on Performance of Neural Networks Using Quadratic Relative Error Cost Function

Ning Zhang,Ye-Shuang Xu,Annan Zhou,Shui-Long Shen

doi:10.1109/access.2019.2930520

Abstract

The performance of neural networks with quadratic cost function (MSE cost function) is analyzed in terms of the adjustment rate of weights and its performance in multi-magnitude data processing using a qualitative mathematical method based on mean squared error. However, neural networks using quadratic cost functions exhibit low-weight updating rates and variations in performances in multi-magnitude data processing. This paper investigates the performance of neural networks using a quadratic relative error cost function (REMSE cost function). Two-node-to-one-node models are built to investigate the performance of the REMSE and MSE cost functions in adjustment rate of weights and multi-magnitude data processing. A three-layer neural network is employed to compare the training and prediction performances of the REMSE cost function and MSE cost function. Three LSTM networks are used to evaluate the differences between REMSE, MSE, and Logcosh in actual applications by learning stress and strain of soil. The results indicate that the REMSE cost function can notably accelerate the adjustment rate of weights and improve the performance of the neural network in small magnitude data regression. The applications of the REMSE cost function are also discussed.

Highlights

Artificial Neural Network (ANN) methods are capable of mapping non-linear relationships between the input and output datasets by involving several sets of hidden layers and weights between them [1], [2]
This study aims to investigate the performance of neural networks with quadratic relative error cost function in data regression
Comparing (8) with (3), we found that the error of each layer in the neural network using the relative mean squared error (REMSE) cost function will be 1/y2i times that using the mean squared error (MSE) cost function

Summary

Introduction

Artificial Neural Network (ANN) methods are capable of mapping non-linear relationships between the input and output datasets by involving several sets of hidden layers and weights between them [1], [2]. A neural network is part of a data processing network, which is composed of layered nodes, weights and activation functions. Data is first imported into the network by input nodes, run through nodes in hidden layers, and output by output nodes. Nodes in hidden layers provide non-linear transfer to data in nodes by activation functions. Weights define the contribution of nodes to the linked nodes. These weights are similar to the weights in other network science, like complex networks [3]–[5]. One of the important applications of an ANN is to fit multidimensional datasets in data processing, instead of using numerous mathematical

Objectives

Discussion

Conclusion