Sparse discriminative Fisher vectors in visual classification

Abstract

Constructing global image representations from local feature descriptors is a common step in most visual classification tasks. Traditionally, the Bag of Features (BoF) representations involving hard vector quantization have been used ubiquitously for such tasks. Recent works have demonstrated superior performance of soft assignments over hard assignments. Fisher vector representations have been shown to outperform other global representations on most benchmark datasets. Fisher vectors (i) use soft assignments, and (ii) reduce information loss due to quantization by capturing the deviations from the mean. However, the Fisher vector representations are huge and the representation size increases linearly with the vocabulary size. Recent findings report that the classification performance of Fisher vectors is proportional to the vocabulary size. Computational and storage requirements, however, discourage the use of arbitrarily large vocabularies. Also, Fisher vectors are not inherently discriminative. In this paper, we devise a novel strategy to compute sparse Fisher representations. This allows us to increase the vocabulary size with little computation and storage overhead and still attain the performance of a larger vocabulary. Further, we describe an approach to encode class-discriminative information in the Fisher vectors. We evaluate our method on four popular datasets. Empirical results show that our representations consistently outperform the traditional Fisher Vector representations and are comparable to the state of art approaches.