Natural Gradient and Multiclass NLDA Networks

Abstract

Natural gradient has been recently introduced as a method to improve the convergence of Multilayer Perceptron (MLP) training [1] as well as that of other neural network type algorithms. The key idea is to recast the training process as a problem in quasi maximum log—likelihood estimation of a certain semipara-metric probabilistic model. This allows the natural introduction of a riemannian metric tensor G in the probabilistic model space. Once G is computed, the “natural” gradient in this setting is \(c G\left( W \right)^{ - 1} \nabla _W e\left( {X,y:W} \right) \) , rather than the ordinary euclidean gradient \( \nabla _W e\left( {X,y;W} \right) \) . Here e(X,y; W) denotes an error function associated to a concrete pattern (X, y) and weight set W. For instance, in MLP training, e(X,y;W) = (y – F(X,W))2/2, with F the MLP transfer function. Viewing (y – F(X, W))2/2 as the log—likelihood of a probability density, the metric tensor is $$ G\left( W \right) = \smallint \smallint \frac{{\partial \log p}} {{\partial W}}\left( {\frac{{\partial \log p}} {{\partial W}}} \right)^t p\left( {X,y;W} \right)dXdy. $$ G(W) is also known as the Fisher Information matrix, as it gives the variance of the Cramer—Rao bound for the optimal W estimator. In this work we shall consider a natural gradient—like training for Non Linear Discriminant Analysis (NLDA) networks, a non—linear extension of Fisher’s well known Linear Discriminant Analysis introduced in [6] (more details below). Instead of following an approach along the previous lines, we observe that (1) can be viewed as the covariance \( G\left( W \right) = E\left[ {\nabla _W e\left( {X,y;W} \right)\nabla _W e\left( {X,y;W} \right)^t } \right] \) of the random vector \( \nabla _W \left( {X,y;W} \right) \) .KeywordsHide LayerLinear Discriminant AnalysisFisher Information MatrixNatural GradientGradient LearningThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.