On Optimization Techniques for the Construction of an Exponential Estimate for Delayed Recurrent Neural Networks

Vasyl Martsenyuk,Mikolaj Karpinski,Stanislaw Rajba

doi:10.3390/sym12101731

Abstract

This work is devoted to the modeling and investigation of the architecture design for the delayed recurrent neural network, based on the delayed differential equations. The usage of discrete and distributed delays makes it possible to model the calculation of the next states using internal memory, which corresponds to the artificial recurrent neural network architecture used in the field of deep learning. The problem of exponential stability of the models of recurrent neural networks with multiple discrete and distributed delays is considered. For this purpose, the direct method of stability research and the gradient descent method is used. The methods are used consequentially. Firstly we use the direct method in order to construct stability conditions (resulting in an exponential estimate), which include the tuple of positive definite matrices. Then we apply the optimization technique for these stability conditions (or of exponential estimate) with the help of a generalized gradient method with respect to this tuple of matrices. The exponential estimates are constructed on the basis of the Lyapunov–Krasovskii functional. An optimization method of improving estimates is offered, which is based on the notion of the generalized gradient of the convex function of the tuple of positive definite matrices. The search for the optimal exponential estimate is reduced to finding the saddle point of the Lagrange function.

Highlights

Breakthrough results in the field of deep machine learning are obtained nowadays using recurrent neural networks (RNN)
The work is devoted to modeling and investigation of the architecture design for the delayed recurrent neural network basing on the delayed differential equations
The usage of discrete and distributed delays makes it possible to model the calculation of the states using internal memory, which corresponds to the artificial recurrent neural network architecture used in the field of deep learning

Summary

Introduction

Breakthrough results in the field of deep machine learning are obtained nowadays using recurrent neural networks (RNN). We use the Hopfield neural network model, which includes a diagonal matrix A with positive entries, that shows the self-connection of the neuron. With the help of differential equations it is possible to explicitly obtain the conditions for stabilization of recurrent neural networks. Work [2] shows the way to construct a RNN of the LSTM type starting from the corresponding model based on differential equations with delay and further through the discretization of the so-called canonical RNN. In the previous works [12,13], indirect method was developed allowing us to get exponential estimates in some general cases of the RNN models It results in the numerical solution of quasipolynomial equation. In order to overcome this shortcoming, here we develop an optimization technique, which is based on the direct method for Liapunov–Krasovskii functionals of the special kind

Exponential Estimate

Optimization Method

Conclusions