Location Property of Convolutional Neural Networks for Image Classification.

Abstract

When doing image classification, the core task of convolutional neural network (CNN)-based methods is to learn better feature representation. Our analysis has shown that a better feature representation in the layer before softmax operation (BSM-layer) means a better feature embedding location that has a larger distance to the separating hyperplane. By defining this property "Location Property" of CNN, the core task of CNN-based methods can be regarded as to find out the optimal feature embedding location in the BSM-layer. In order to achieve this, in this work, we first propose two feature embedding directions, principal embedding direction (PE-direction) and secondary embedding direction (SE-direction). And then, we further propose a loss-based optimization framework, location property loss (LP-loss), which can make feature representation move in the PE-direction and the SE-direction simultaneously during the training phase. LP-loss consists of two parts, LPPE and LPSE, where LPPE focuses on PE-direction, and LPSE focuses on SE-direction. Any loss function focusing on these two embedding directions can be chosen as LPPE and LPSE. Based on the analysis that softmax, L-softmax, and AM softmax can make the feature representation move in PE-direction to a different extent, any of them can be chosen as LPPE. Since there is no existing works can fulfill the purpose of LPSE, a novel loss, secondary optimal feature plane loss (S-OFP loss), is developed. S-OFP loss is designed to make feature representations belonging to the same category embed onto their corresponding S-OFP. It is proved that S-OFP loss is the optimal feature plane in the SE-direction. Experiments are done with shallow, moderate, and deep models on four benchmark data sets, including the MNIST, SVHN, CIFAR-10, and CIFAR-100, and results demonstrate that CNN models can obtain remarkable performance improvements with LPsoftmax, S-OFP and LPAM softmax, S-OFP, which verify the effectiveness of location property.