Tucker network: Expressive power and comparison

Abstract

Deep neural networks have achieved great success in solving many machine learning and computer vision problems. In this paper, we propose a deep neural network called the Tucker network derived from the Tucker format and analyze its expressive power. The results demonstrate that the Tucker network has exponentially higher expressive power than the shallow network. In other words, a shallow network with an exponential width is required to realize the same score function as that computed by the Tucker network. Moreover, we discuss the expressive power between the hierarchical Tucker tensor network (HT network) and the proposed Tucker network. To generalize the Tucker network into a deep version, we combine the hierarchical Tucker format and Tucker format to propose a deep Tucker tensor decomposition. Its corresponding deep Tucker network is presented. Experiments are conducted on three datasets: MNIST, CIFAR-10 and CIFAR-100. The results experimentally validate the theoretical results and show that the Tucker network and deep Tucker network have better performance than the shallow network and HT network.