Covariance Pooling Research Articles

Enhancing breast cancer histopathological image classification using attention-based high order covariance pooling

Enhancing breast cancer histopathological image classification using attention-based high order covariance pooling

One-Dimensional-Mixed Convolution Neural Network and Covariance Pooling Model for Mineral Mapping of Porphyry Copper Deposit Using PRISMA Hyperspectral Data

Mapping distribution of alterations around porphyry copper deposits (PCDs) greatly affects mineral exploration. Diverse geological processes generate irregular alteration patterns with diverse spectral characteristics in mineral deposits. Applying remotely sensed hyperspectral images (HSIs) is an appealing technology for geologic surveyors to generate alteration maps. Conventional methods mainly use shallow spectral absorption features to discriminate minerals and cannot extract their important spectral information. Deep neural networks with nonlinear layers can evoke the deep spectral and spatial information of HSIs. Deep learning???based methods include fully connected neural networks, convolutional neural networks, and hybrid convolutional networks like mixed convolution neural network and covariance pooling (MCNN‐CP) algorithms. However, each has its advantages and limitations. To significantly avoid losing important spectral features, we proposed a new method by fusing a one‐dimensional convolutional neural network (1D‐CNN) with MCNN‐CP (1D‐MCNN‐CP), achieving an overall accuracy (97.44%) of mineral mapping from PRISMA HSIs. This research deduced that 1D‐MCNN‐CP improved performance and reduced misclassification errors among minerals sharing similar spectral features.

SSTrack: An Object Tracking Algorithm Based on Spatial Scale Attention

The traditional Siamese object tracking algorithm uses a convolutional neural network as the backbone and has achieved good results in improving tracking precision. However, due to the lack of global information and the use of spatial and scale information, the accuracy and speed of such tracking algorithms still need to be improved in complex environments such as rapid motion and illumination variation. In response to the above problems, we propose SSTrack, an object tracking algorithm based on spatial scale attention. We use dilated convolution branch and covariance pooling to build a spatial scale attention module, which can extract the spatial and scale information of the target object. By embedding the spatial scale attention module into Swin Transformer as the backbone, the ability to extract local detailed information has been enhanced, and the success rate and precision of tracking have been improved. At the same time, to reduce the computational complexity of self-attention, Exemplar Transformer is applied to the encoder structure. SSTrack achieved 71.5% average overlap (AO), 86.7% normalized precision (NP), and 68.4% area under curve (AUC) scores on the GOT-10k, TrackingNet, and LaSOT. The tracking speed reached 28fps, which can meet the need for real-time object tracking.

Hyperspectral classification algorithm based on covariance pooling and cross scale feature extraction

Hyperspectral classification algorithm based on covariance pooling and cross scale feature extraction

RD-OpenMax: Rethinking OpenMax for Robust Realistic Open-Set Recognition.

Open-set recognition (OSR) toward a practical open-world setting has attracted increasing research attention in recent years. However, existing OSR settings are either too idealized or focus on specific scenes such as long-tailed distribution and few-shot samples, which fail to capture the complexity of real-world scenarios. In this article, we propose a realistic OSR (ROSR) setting that covers a diverse range of challenging and real-world scenarios, including fine-grained cases with strong semantic correlation and a large number of species, few-shot samples, long-tailed sample distribution, dynamic inputs (e.g., images, spatio-temporal, and multimodal signals) and cross-domain adaptation. In particular, we rethink the simple and basic OpenMax for the ROSR setting and introduce a novel method, regularized discriminative OpenMax (RD-OpenMax), to handle the challenges in the ROSR setting. RD-OpenMax improves upon the basic OpenMax approach by introducing a covariance attention-based covariance pooling (CACP) module as a global aggregation step before the deep architecture's classifier. This module explores rich statistical information on features and provides discriminative distance scores for OpenMax. To address the instability of extreme value theory (EVT) estimation due to insufficient training samples under few-shot and long-tailed scenarios, we propose a regularized EVT (REVT) method based on Monte Carlo sampling to recalibrate the distribution of distance scores. As such, our RD-OpenMax performs a REVT model of distance scores generated by discriminative CACP representations to distinguish known classes and recognize unknown ones effectively and robustly. Extensive experiments are conducted on more than ten visual benchmarks across several scenarios, and the empirical comparisons show that the ROSR setting challenges existing state-of-the-art OSR approaches. Moreover, our RD-OpenMax clearly outperforms its counterparts under the ROSR setting while performing favorably against state-of-the-arts under the traditional OSR setting.

CCP-GNN: Competitive Covariance Pooling for Improving Graph Neural Networks.

Graph neural networks (GNNs) have advanced graph classification tasks, where a global pooling to generate graph representations by summarizing node features plays a critical role in the final performance. Most of the existing GNNs are built with a global average pooling (GAP) or its variants, which however, take no full consideration of node specificity while neglecting rich statistics inherent in node features, limiting classification performance of GNNs. Therefore, this article proposes a novel competitive covariance pooling (CCP) based on observation of graph structures, i.e., graphs generally can be identified by a (small) key part of nodes. To this end, our CCP generates node-level second-order representations to explore rich statistics inherent in node features, which are fed to a competitive-based attention module for effectively discovering key nodes through learning node weights. Subsequently, our CCP aggregates node-level second-order representations in conjunction with node weights by summation to produce a covariance representation for each graph, while an iterative matrix normalization is introduced to consider geometry of covariances. Note that our CCP can be flexibly integrated with various GNNs (namely CCP-GNN) to improve the performance of graph classification with little computational cost. The experimental results on seven graph-level benchmarks show that our CCP-GNN is superior or competitive to state-of-the-arts. Our code is available at https://github.com/Jillian555/CCP-GNN.

Towards A Deeper Understanding of Global Covariance Pooling in Deep Learning: An Optimization Perspective.

Global covariance pooling (GCP) as an effective alternative to global average pooling has shown good capacity to improve deep convolutional neural networks (CNNs) in a variety of vision tasks. Although promising performance, it is still an open problem on how GCP (especially its post-normalization) works in deep learning. In this paper, we make the effort towards understanding the effect of GCP on deep learning from an optimization perspective. Specifically, we first analyze behavior of GCP with matrix power normalization on optimization loss and gradient computation of deep architectures. Our findings show that GCP can improve Lipschitzness of optimization loss and achieve flatter local minima, while improving gradient predictiveness and functioning as a special pre-conditioner on gradients. Then, we explore the effect of post-normalization on GCP from the model optimization perspective, which encourages us to propose a simple yet effective normalization, namely DropCov. Based on above findings, we point out several merits of deep GCP that have not been recognized previously or fully explored, including faster convergence, stronger model robustness and better generalization across tasks. Extensive experimental results using both CNNs and vision transformers on diversified vision tasks provide strong support to our findings while verifying the effectiveness of our method.

Comprehensive receptive field adaptive graph convolutional networks for action recognition

Action recognition has shown a broad prospect for application in human–computer interaction and autonomous vehicles. According to skeleton data being lightweight and robust to environmental disturbance, skeleton-based methods are increasingly used in action recognition. Existing Graph Convolutional Networks (GCNs) have limitations such as Global Average Pooling (GAP) causing information oversmoothing, attention mechanism increasing computational complexity, and fixed spatial configuration partition strategy limiting flexibility. Our model overcomes these limitations by incorporating Temporal Covariance Pooling (TCP) for better feature utilization, Sample-Shared attention for weight sharing, and a Comprehensive Receptive Field strategy for optimal joint connections. Experiments show that the recognition accuracy of our model outperforms baseline 2s-AGCN by 1.1 % and 0.7 % under X-Sub and X-View for NTU RGB + D dataset, 1.8 % and 1.9 % in X-Sub and X-Set for NTU RGB + D 120 dataset, and 1.6 % on Kinetics-Skeleton dataset.

Blind super-resolution model based on unsupervised degenerate indication learning

Blind super-resolution of images based on contrast learning has achieved better performance, which can distinguish between different degraded information. However, there is no significant improvement in the processing of image super-resolution. To further improve the performance of image blind super-resolution, a new degenerate higher-order attention network (DHAN) is designed for image super-resolution, in which a higher-order channel attention mechanism is introduced to the framework. Meanwhile, it also introduces normalized variance and global covariance pooling, which can provide higher-order feature information and more discriminative feature information. Comparing the experimental results, it can be found that the proposed higher-order attention network (DHAN) is able to distinguish various complex degradation patterns and thus obtain accurate degradation information. Numerous experiments have shown that higher-order attention network (DHAN) achieves good visual results in the image blind super-resolution task.

Orthogonal SVD Covariance Conditioning and Latent Disentanglement.

Inserting an SVD meta-layer into neural networks is prone to make the covariance ill-conditioned, which could harm the model in the training stability and generalization abilities. In this article, we systematically study how to improve the covariance conditioning by enforcing orthogonality to the Pre-SVD layer. Existing orthogonal treatments on the weights are first investigated. However, these techniques can improve the conditioning but would hurt the performance. To avoid such a side effect, we propose the Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR). The effectiveness of our methods is validated in two applications: decorrelated Batch Normalization (BN) and Global Covariance Pooling (GCP). Extensive experiments on visual recognition demonstrate that our methods can simultaneously improve covariance conditioning and generalization. The combinations with orthogonal weight can further boost the performance. Moreover, we show that our orthogonality techniques can benefit generative models for better latent disentanglement through a series of experiments on various benchmarks. Code is available at: https://github.com/KingJamesSong/OrthoImproveCond.

Fast Differentiable Matrix Square Root and Inverse Square Root.

Computing the matrix square root and its inverse in a differentiable manner is important in a variety of computer vision tasks. Previous methods either adopt the Singular Value Decomposition (SVD) to explicitly factorize the matrix or use the Newton-Schulz iteration (NS iteration) to derive the approximate solution. However, both methods are not computationally efficient enough in either the forward pass or the backward pass. In this paper, we propose two more efficient variants to compute the differentiable matrix square root and the inverse square root. For the forward propagation, one method is to use Matrix Taylor Polynomial (MTP), and the other method is to use Matrix Padé Approximants (MPA). The backward gradient is computed by iteratively solving the continuous-time Lyapunov equationusing the matrix sign function. A series of numerical tests show that both methods yield considerable speed-up compared with the SVD or the NS iteration. Moreover, we validate the effectiveness of our methods in several real-world applications, including de-correlated batch normalization, second-order vision transformer, global covariance pooling for large-scale and fine-grained recognition, attentive covariance pooling for video recognition, and neural style transfer. The experiments demonstrate that our methods can also achieve competitive and even slightly better performances. Code is available at https://github.com/KingJamesSong/FastDifferentiableMatSqrt.

Multi-Scale Feature Fusion of Covariance Pooling Networks for Fine-Grained Visual Recognition.

Multi-scale feature fusion techniques and covariance pooling have been shown to have positive implications for completing computer vision tasks, including fine-grained image classification. However, existing algorithms that use multi-scale feature fusion techniques for fine-grained classification tend to consider only the first-order information of the features, failing to capture more discriminative features. Likewise, existing fine-grained classification algorithms using covariance pooling tend to focus only on the correlation between feature channels without considering how to better capture the global and local features of the image. Therefore, this paper proposes a multi-scale covariance pooling network (MSCPN) that can capture and better fuse features at different scales to generate more representative features. Experimental results on the CUB200 and MIT indoor67 datasets achieve state-of-the-art performance (CUB200: 94.31% and MIT indoor67: 92.11%).

Improving Pre-Training and Fine-Tuning for Few-Shot SAR Automatic Target Recognition

SAR-ATR (synthetic aperture radar-automatic target recognition) is a hot topic in remote sensing. This work suggests a few-shot target recognition approach (FTL) based on the concept of transfer learning to accomplish accurate target recognition of SAR images in a few-shot scenario since the classic SAR ATR method has significant data reliance. At the same time, the strategy introduces a model distillation method to improve the model’s performance further. This method is composed of three parts. First, the data engine, which uses the style conversion model and optical image data to generate image data similar to SAR style and realize cross-domain conversion, can effectively solve the problem of insufficient training data of the SAR image classification model. Second is model training, which uses SAR image data sets to pre-train the model. Here, we introduce the deep Brownian distance covariance (Deep BDC) pooling layer to optimize the image feature representation so that the model can learn the image representation by measuring the difference between the joint feature function of the embedded feature and the edge product. Third, model fine-tuning, which freezes the model structure, except the classifier, and fine-tunes it by using a small amount of novel data. The knowledge distillation approach is also introduced simultaneously to train the model repeatedly, sharpen the knowledge, and enhance model performance. According to experimental results on the MSTAR benchmark dataset, the proposed method is demonstrably better than the SOTA method in the few-shot SAR ATR issue. The recognition accuracy is about 80% in the case of 10-way 10-shot.

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

PiCovS: Pixel-Level With Covariance Pooling Feature and Superpixel-Level Feature Fusion for Hyperspectral Image Classification

PiCovS: Pixel-Level With Covariance Pooling Feature and Superpixel-Level Feature Fusion for Hyperspectral Image Classification

Universal Image Steganalysis Based on Convolutional Neural Network with Global Covariance Pooling

Universal Image Steganalysis Based on Convolutional Neural Network with Global Covariance Pooling

Breast cancer histopathology image classification based on dual-stream high-order network

The early diagnosis of breast cancer using pathological images is of the vital importance. Recently, breast cancer histopathology image classification methods based on convolution neural network (CNN) are constantly innovating with the development of computer-aided diagnosis technology. To obtain pathological tissue features with more discriminant presentation capability for classification, this work proposes a novel dual-stream high-order breast cancer pathological image classification network named DsHoNet. To be precise, a shallow network composed of six convolution layers is built as the backbone of the dual-stream network firstly, in which one stream utilizes batch normalization (BN) layer to retain the original feature information with clearer feature distribution, while another stream introduces the Ghost module to extract richer supplementary features by utilizing a series of linear transformations. Then, outputs of the two streams are further enhanced via a covariance pooling layer to achieve more powerful deep high-order statistic features for classification. Extensive evaluation experiments carried out on the public BreakHis dataset demonstrate that the optimal recognition rates of DsHoNet are 99.01% and 99.25% respectively at the image-level and patient-level, performing favorably against its counterparts.

Remote sensing scene classification with multi-spatial scale frequency covariance pooling

Remote sensing scene classification with multi-spatial scale frequency covariance pooling

Second-order asymmetric convolution network for breast cancer histopathology image classification.

Recently, convolutional neural networks (CNNs) have been widely utilized for breast cancer histopathology image classification. Besides, research works have also convinced that deep high-order statistic models obviously outperform corresponding first-order counterparts in vision tasks. Inspired by this, we attempt to explore global deep high-order statistics to distinguish breast cancer histopathology images. To further boost the classification performance, we also integrate asymmetric convolution into the second-order network and propose a novel second-order asymmetric convolution network (SoACNet). SoACNet adopts a series of asymmetric convolution blocks to replace each stand square-kernel convolutional layer of the backbone architecture, followed by a global covariance pooling to compute second-order statistics of deep features, leading to a more robust representation of histopathology images. Extensive experiments on the public BreakHis dataset demonstrate the effectiveness of SoACNet for breast cancer histopathology image classification, which achieves competitive performance with the state-of-the-arts.

Diagnosis of schizophrenia with functional connectome data: a graph-based convolutional neural network approach

Previous deep learning methods have not captured graph or network representations of brain structural or functional connectome data. To address this, we developed the BrainNet-Global Covariance Pooling-Attention Convolutional Neural Network (BrainNet-GA CNN) by incorporating BrainNetCNN and global covariance pooling into the self-attention mechanism. Resting-state functional magnetic resonance imaging data were obtained from 171 patients with schizophrenia spectrum disorders (SSDs) and 161 healthy controls (HCs). We conducted an ablation analysis of the proposed BrainNet-GA CNN and quantitative performance comparisons with competing methods using the nested tenfold cross validation strategy. The performance of our model was compared with competing methods. Discriminative connections were visualized using the gradient-based explanation method and compared with the results obtained using functional connectivity analysis. The BrainNet-GA CNN showed an accuracy of 83.13%, outperforming other competing methods. Among the top 10 discriminative connections, some were associated with the default mode network and auditory network. Interestingly, these regions were also significant in the functional connectivity analysis. Our findings suggest that the proposed BrainNet-GA CNN can classify patients with SSDs and HCs with higher accuracy than other models. Visualization of salient regions provides important clinical information. These results highlight the potential use of the BrainNet-GA CNN in the diagnosis of schizophrenia.