Improved deep neural network performance under dynamic programming mode

Abstract

For Deep Neural Networks (DNN), the standard gradient-based algorithms may not be efficient because of the raised computational expense resulting from the increase in the number of layers. This paper offers an alternative to the classic training solutions: an in-depth study to find conditions under which the underlying Artificial Neural Networks ANN minimisation problem can be addressed from a Dynamic Programming (DP) perspective. Specifically, we prove that any ANN with monotonic activation is separable when regarded as a parametric function. Particularly, when the ANN is viewed as a network representation of a dynamical system (as a coupled cell network), we also prove that the transmission-of-signal law is separable provided the activation function is a monotone non-decreasing function. This strategy may have a positive impact on the performance of ANNs by improving their learning accuracy, particularly for DNNs. For our purposes, ANNs are also viewed as universal approximators of continuous functions and as abstract compositions of an even number of functions. This broader representation makes it easier to analyse them from many other perspectives (universal approximation issues, inverse problem solving) leading to a general improvement in knowledge on NNs and their performance.