Heterogeneous Acceleration of Hybrid PSO-QN Algorithm for Neural Network Training

Shun Yan,Liang Han,Jiajun Li,Qiang Liu

doi:10.1109/access.2019.2951710

Abstract

Artificial neural network (ANN) has successfully provided solutions to many practical problems. One of the difficulties in training ANNs is finding the ideal solution to the network weights quickly. This paper designs an implementation of the hybrid particle swarm optimization (PSO) and quasi-Newton (QN) algorithm on CPU-GPU platform using OpenCL to accelerate ANN training. The PSO-QN implementation combines the strength of the PSO algorithm in global search and the advantage of the QN algorithm in fast convergence rate. A configurable parallel line search implementation and an efficient parallel reinitialization implementation are proposed to improve the performance and reduce data transmission. Experiments show the PSO-QN hybrid parallel implementation on CPU-GPU platform can achieve up to $362\times$ and $8.9\times$ acceleration compared with the C++ implementations of PSO and BFGS-QN on CPU, respectively. Compared with the PSO and BFGS-QN parallel implementations, the training loss of the PSO-QN hybrid implementation at given training time is reduced by 15.16% and 3.97%, and the testing error is reduced by 13.66% and 6.86%, respectively.

Highlights

Research on artificial neural network (ANN) has made great progress, and provides solutions for many practical problems in early detection [1], microwave circuit design [2], energy expenditure estimation [3], etc.ANN still faces difficulties in practical applications, one of which is training
To deal with the local optimum problem, global optimization algorithms like particle swarm optimization (PSO) algorithm [6] are used for ANN training
Compared with the PSO and BFGS-QN parallel implementations, the training loss of the PSO-QN hybrid implementation at given training time is reduced by 15.16% and 3.97%, and the testing error is reduced by 13.66% and 6.86%, respectively

Summary

Introduction

Research on artificial neural network (ANN) has made great progress, and provides solutions for many practical problems in early detection [1], microwave circuit design [2], energy expenditure estimation [3], etc.ANN still faces difficulties in practical applications, one of which is training. Research on artificial neural network (ANN) has made great progress, and provides solutions for many practical problems in early detection [1], microwave circuit design [2], energy expenditure estimation [3], etc. Each iteration of the algorithm is carried out along the direction in which the loss function drops the fastest. This algorithm is easy to fall into local optimum and has slow convergence rate [5]. To deal with the local optimum problem, global optimization algorithms like particle swarm optimization (PSO) algorithm [6] are used for ANN training. As a population based stochastic optimization technique, the PSO algorithm has global search ability which increases the probability of finding the global optimal solution. The PSO algorithm requires a large number of particles to participate in the computation, The associate editor coordinating the review of this manuscript and approving it for publication was Aysegul Ucar

Methods

Results

Conclusion