VecHGrad for Solving Accurately Tensor Decomposition

Abstract

Tensor decomposition is a collection of factorization techniques for multidimensional arrays. Today’s data sets, because of their size, require tensor decomposition involving factorization with multiple matrices and diagonal tensors such as DEDICOM or PARATUCK2. Traditional tensor resolution algorithms such as Stochastic Gradient Descent (SGD) or Non-linear Conjugate Gradient descent (NCG), cannot be easily applied to these types of tensor decomposition or often lead to poor accuracy at convergence. We propose a new resolution algorithm, VecHGrad, for accurate and efficient stochastic resolution over all existing tensor decomposition. VecHGrad relies on the gradient, an Hessian-vector product, and an adaptive line search, to ensure the convergence during optimization. Our experiments on five popular data sets with the state-of-the-art deep learning gradient optimizers show that VecHGrad is capable of converging considerably faster because of its superior convergence rate per step. VecHGrad targets as well deep learning optimizer algorithms. The experiments are performed for various tensor decomposition, including CP, DEDICOM, and PARATUCK2. Although it involves an Hessian-vector update rule, VecHGrad’s runtime is similar in practice to that of gradient methods such as SGD, Adam, or RMSProp.