Covariance Pooling Research Articles

Skip-Connected Covariance Network for Remote Sensing Scene Classification

This paper proposes a novel end-to-end learning model, called skip-connected covariance (SCCov) network, for remote sensing scene classification (RSSC). The innovative contribution of this paper is to embed two novel modules into the traditional convolutional neural network (CNN) model, i.e., skip connections and covariance pooling. The advantages of newly developed SCCov are twofold. First, by means of the skip connections, the multi-resolution feature maps produced by the CNN are combined together, which provides important benefits to address the presence of large-scale variance in RSSC data sets. Second, by using covariance pooling, we can fully exploit the second-order information contained in such multi-resolution feature maps. This allows the CNN to achieve more representative feature learning when dealing with RSSC problems. Experimental results, conducted using three large-scale benchmark data sets, demonstrate that our newly proposed SCCov network exhibits very competitive or superior classification performance when compared with the current state-of-the-art RSSC techniques, using a much lower amount of parameters. Specifically, our SCCov only needs 10% of the parameters used by its counterparts.

Deep Covariance Estimation Hashing

Deep hashing, the combination of advanced convolutional neural networks and efficient hashing, has recently achieved impressive performance for image retrieval. However, state-of-the-art deep hashing methods mainly focus on constructing hash function, loss function and training strategies to preserve semantic similarity. For the fundamental image characteristics, they depend heavily on the first-order convolutional feature statistics, failing to take their global structure into consideration. To address this problem, we present a deep covariance estimation hashing (DCEH) method with robust covariance form to improve hash code quality. The core of DCEH involves covariance pooling as deep hashing representation, performing global pairwise feature interactions. The covariance pooling can capture richer statistic information of deep convolutional features and produce more informative global representations.Due to convolutional features are usually high dimension and small sample size, we estimate robust covariance by shrinking its eigenvalues using power normalization, forming an independent structural layer. Then the structural layer is embedded into deep hashing paradigm in an end-to-end learning manner. Extensive experiments on three benchmarks show that the proposed DCEH outperforms its counterparts and achieves superior performance.

Data Augmentation and Second-Order Pooling for Facial Expression Recognition

Facial expression is the main medium of information transmission in human communication, playing an important role in human’s daily life. Facial expression recognition is still challenging due to the various obstacle, illumination, and posture. However, most of the existing works focus on deeper or wider network structures and rarely explores the high-level feature statistics. In this paper, we propose a second-order pooling convolution neural network to explore the correlation information between the facial features after deep network learning. At the final stage of the network, we add a new covariance pooling layer to replace the first-order pooling of standard convolution networks. In the pooling layer of covariance, the Newton iteration method is used to approximate the square root instead of EIG or SVD, which makes it more suitable for GPU. Due to the small amount of facial expression data, this paper uses different data augmentation methods to increase the amount of training data and improve the generalization ability of the model. The proposed method, data augmentation and second-order pooling (DASOP), was evaluated on the real-world affective faces database (RAFDB) and the static facial expressions in the wild (SFEW), yielding correct rates of 88.625% and 59.518%, respectively. We achieve state-of-the-art performance superior to existing methods.

Remote Sensing Scene Classification Using Multilayer Stacked Covariance Pooling

This paper proposes a new method, called multilayer stacked covariance pooling (MSCP), for remote sensing scene classification. The innovative contribution of the proposed method is that it is able to naturally combine multilayer feature maps, obtained by pretrained convolutional neural network (CNN) models. Specifically, the proposed MSCP-based classification framework consists of the following three steps. First, a pretrained CNN model is used to extract multilayer feature maps. Then, the feature maps are stacked together, and a covariance matrix is calculated for the stacked features. Each entry of the resulting covariance matrix stands for the covariance of two different feature maps, which provides a natural and innovative way to exploit the complementary information provided by feature maps coming from different layers. Finally, the extracted covariance matrices are used as features for classification by a support vector machine. The experimental results, conducted on three challenging data sets, demonstrate that the proposed MSCP method can not only consistently outperform the corresponding single-layer model but also achieve better classification performance than other pretrained CNN-based scene classification methods.

Tracking and recognition face in videos with incremental local sparse representation model

This paper addresses the problem of tracking and recognizing faces via incremental local sparse representation. First a robust face tracking algorithm is proposed via employing local sparse appearance and covariance pooling method. In the following face recognition stage, with the employment of a novel template update strategy, which combines incremental subspace learning, our recognition algorithm adapts the template to appearance changes and reduces the influence of occlusion and illumination variation. This leads to a robust video-based face tracking and recognition with desirable performance. In the experiments, we test the quality of face recognition in real-world noisy videos on YouTube database, which includes 47 celebrities. Our proposed method produces a high face recognition rate at 95% of all videos. The proposed face tracking and recognition algorithms are also tested on a set of noisy videos under heavy occlusion and illumination variation. The tracking results on challenging benchmark videos demonstrate that the proposed tracking algorithm performs favorably against several state-of-the-art methods. In the case of the challenging dataset in which faces undergo occlusion and illumination variation, and tracking and recognition experiments under significant pose variation on the University of California, San Diego (Honda/UCSD) database, our proposed method also consistently demonstrates a high recognition rate.

Covariance pooling and stabilization for classification

It is well known that the quadratic discriminant rule (QD) is optimal for large, normal training sets in the sense of minimizing the overall misclassification rate. However, when the size of the training set is small compared to the number of variables, the performance of QD is degraded because it uses unstable sample mean vectors and covariance matrices. Friedman [8] and Greene and Rayens [9] recently proposed different methods for addressing the problem of unstable covariance matrices. This article details a critical comparison of these two methods, finding important strengths and weaknesses in both. In addition, a third discriminant rule which combines ideas from both of the aforementioned methods is developed and included in the comparisons.