Attention Inspired Network: Steep learning curve in an invariant pattern recognition model

Abstract

Hubel and Wiesel’s study about low areas of the visual cortex (VC) inspired deep models for invariant pattern recognition. In such models, simple and complex layers alternate local feature extraction with subsampling to add invariance to distortion or transformations. However, it was shown that to tolerate large changes between examples of the same category, the subsampling operation has to discard so much information that the model loses the capability to discriminate between categories. So, in practice, small changes are tolerated by these layers and, afterwards, a powerful classifier is introduced to do the rest. By incorporating insights from higher areas of the VC, we add to the already used retinotopic step an object-centered step which increases invariance capabilities without losing so much information. By doing so, we reduce the need for a powerful, data hungry classification layer and, thus, are able to introduce a simple classification mechanism which is based on selective attention. The resulting model is tested with an invariant pattern recognition task in the MNIST and ETL-1 datasets. We verify that the model is able to achieve better accuracies with less training examples. More specifically, on the MNIST test set, the model achieves a 100% accuracy when trained with little more than 10% of the training set.