Learning Deep Co-Occurrence Features

Abstract

We exploit the computational capability of deep convolutional neural network (CNN) architecture and the natural interpretability of the co-occurrence matrix (CM) to learn deep co-occurrence features (DCOFs). The DCOFs represent the statistics of the co-occurrences of pixels thus overcoming the black box nature of traditional deep representation learning while at the same time solving the inherent computational difficulty of CM. We propose a parametric co-occurrence matrix (PCM) model to approximate the CM with multivariate Gaussian functions, and have developed three approaches to decomposing the PCM model into linear and nonlinear operations such that the model can be easily implemented using standard CNN operations and to learn the DCOFs of arbitrary shapes. The CNN implementation of the PCM model, termed PCMCNN, can be used as a standard plugin module of a deep learning system and adaptively learns the DCOFs for downstream applications. We demonstrate the broad applicability of the DCOFs and their effectiveness in fine-grained image classification tasks such as texture classification and GAN (generative adversarial network) image detection. The introduction of the PCMCNN module makes it much more compact and efficient than conventional implementations of deep learning models, achieving comparable classification performances to state of the art methods on a variety of benchmarking datasets with models that are more than 30 folds smaller and 11 times less complex. The small model size for learning the DCOFs makes the new method particularly effective for few shot classification of large number of texture categories where the small number of training samples can easily cause traditional deep learning models to overfit. This work shows the potential benefits of combining the principles of traditional handcrafted features and deep representation learning to take advantage of both for advancing state of the art.