Revisiting hierarchy: Deep learning with orthogonally constrained prior for classification

Abstract

• A novel approach with hierarchical constraints over labels under deep learning framework. • Introduce weight decomposition to model parameter sharing along the root of the hierarchy to leafs. • Incorporate the orthogonal restrictions among adjacent nodes in the hierarchy. • Compared to competitive baselines, our approach yields significant better results when a few training data are available. Deep learning has attracted significant attention for its applications to a variety of classification problems, such as handwritten recognition, image classification and document categorization. One reason behind the success of deep learning can be attributed to its strong representation power with multiple layers of hidden variables. However, complex models are often encompassed with overfitting problems when limited training data is available. In this paper, we are interested in deep learning for classification with prior, where the set of labels are expressed in a hierarchy. In particular, we attempt to leverage knowledge transfer and parameter sharing among classes. We introduce the orthogonally constrained prior into deep learning, which exploits the following information among different classes: (1) introduce weight decomposition to model parameter sharing along the path from the root to leafs; (2) incorporate the orthogonal restrictions among adjacent nodes in the hierarchy. We test our method on challenge datasets in the case of a few training examples available, and show promising results compared to support vector machines and other deep learning methods .