Local Linear Approximation Algorithm for Neural Network

Mudong Zeng,Agus Sudjianto,Yujie Liao,Runze Li

doi:10.3390/math10030494

Mudong Zeng, Agus Sudjianto + Show 2 more

Open Access

https://doi.org/10.3390/math10030494

Copy DOI

Abstract

This paper aims to develop a new training strategy to improve efficiency in estimation of weights and biases in a feedforward neural network (FNN). We propose a local linear approximation (LLA) algorithm, which approximates ReLU with a linear function at the neuron level and estimate the weights and biases of one-hidden-layer neural network iteratively. We further propose the layer-wise optimized adaptive neural network (LOAN), in which we use the LLA to estimate the weights and biases in the LOAN layer by layer adaptively. We compare the performance of the LLA with the commonly-used procedures in machine learning based on seven benchmark data sets. The numerical comparison implies that the proposed algorithm may outperform the existing procedures in terms of both training time and prediction accuracy.

Highlights

Despite that neural networks have many successful applications, there is a longstanding challenge of training feedforward neural network (FNN) models efficiently to strike a balance between training time and prediction performance
The work in this paper aims to address the problem from a different point of view, and develop a reliable training algorithm called Local Linear Approximation (LLA) Algorithm
The results indicate that LLA has a faster convergence rate and outperforms existing optimization methods and machine learning models in terms of prediction on both training and testing data

Summary

Introduction

Despite that neural networks have many successful applications, there is a longstanding challenge of training feedforward neural network (FNN) models efficiently to strike a balance between training time and prediction performance. In order to reduce total training iterations and training time, while achieving high generalization ability, many studies have developed different efficient training algorithms for neural networks, among which linear approximation of the neural network received lots of attentions. Douglas and Meng [4], Singhal and Wu [5] and Stan and Kamen [6] developed linearized least squares using Kalman recursion to update weight parameters. They demonstrated increased training efficiency of this technique compared to gradient descent. These methods lacked robustness in terms of model accuracy

Objectives

Methods

Findings

Conclusion