An Orthogonal Classification Layer with Kasami Sequences for Discriminative Feature Learning in Neural Networks

Abstract

This paper proposes a novel Orthogonal Classification Layer (OCL) utilizing Kasami sequences for neural networks trained for classification problems. OCL consists of a fully connected layer with fixed orthogonal weights and zero biases for all of its output neurons. Attaching the OCL to the end of any neural network encourages the network to generate a unique latent representation (orthogonal code) at its last hidden layer for each data class. This is achieved by associating and fixing the weights for each of the output neurons to a unique code from a set of orthogonal sequences, such as Kasami. When networks use OCL the latent representations they learn for each data class at the end of the network converge to values equivalent to the fixed weights of the OCL. Auto-correlation and cross-correlation properties of orthogonal codes maximize the output of the correct class and increase its separation from the outputs of all other classes. Therefore, networks trained with OCL benefit from a wider classification decision margin than networks without OCL. Moreover, the feature sets extracted by the network for different data classes are well separated and more amenable to human interpretation. The implementation of OCL is simple and OCL can easily be integrated into any neural network architecture without a need for network architecture modifications or changes to the training scheme (for example optimization, normalization, or regularization) adopted in the original network architecture. The computational and memory cost required for the OCL is lower than conventional classification layers since the weights are fixed and no gradient is required. Though simple and practical the proposed OCL enhances learning of discriminative latent representations, generates explainable features, and provides high classification accuracy. We demonstrate this through a set of evaluation experiments comparing the performance of equivalent networks with and without OCL.