Non-local Attention Research Articles

Siamese Implicit Region Proposal Network With Compound Attention for Visual Tracking.

Recently, siamese-based trackers have achieved significant successes. However, those trackers are restricted by the difficulty of learning consistent feature representation with the object. To address the above challenge, this paper proposes a novel siamese implicit region proposal network with compound attention for visual tracking. First, an implicit region proposal (IRP) module is designed by combining a novel pixel-wise correlation method. This module can aggregate feature information of different regions that are similar to the pre-defined anchor boxes in Region Proposal Network. To this end, the adaptive feature receptive fields then can be obtained by linear fusion of features from different regions. Second, a compound attention module including a channel and non-local attention is raised to assist the IRP module to perform a better perception of the scale and shape of the object. The channel attention is applied for mining the discriminative information of the object to handle the background clutters of the template, while non-local attention is trained to aggregate the contextual information to learn the semantic range of the object. Finally, experimental results demonstrate that the proposed tracker achieves state-of-the-art performance on six challenging benchmark tests, including VOT-2018, VOT-2019, OTB-100, GOT-10k, LaSOT, and TrackingNet. Further, our obtained results demonstrate that the proposed approach can be run at an average speed of 72 FPS in real time.

A CAM-Based Weakly Supervised Method for Surface Defect Inspection

Product surface defect detection is vital to ensure the quality, efficiency, and reliability of industrial production. Deep-learning-based product surface defect inspection algorithms have been gradually used because of their higher detection accuracy and better generalization performance. However, the current deep-learning-based algorithms require a large amount of training samples and high-cost manual annotation work, which is inefficient and costly. In this article, we propose a weakly supervised defect segmentation algorithm of image-level labels based on a classification activation map (CAM). First, we use a Siamese network to narrow the gap between image-level and pixel-level supervision. Then, three modules are improved to enhance the inspection performance, i.e., auto-focused subregion loss, max-pooling-based nonlocal attention, and log summation exponential global pooling, which are used to boost the segmentation without additional computation complexity. To evaluate the performance of the proposed method, we conduct comparison experiments on two public datasets: Deutsche Arbeitsgemeinschaft fuer Mustererkennung (DAGM) and KolektorSDD. The experimental results showed that the proposed method is superior and generalized than state-of-the-art weakly supervised methods. Furthermore, our method outperforms some early fully supervised segmentation algorithms.

A Non-Local Attention Feature Fusion Network for Multiscale Object Detection

Feature pyramid networks (FPN) provide typical architectures for building learning networks with advanced semantic features, which are essential for object recognition at different scales. However, FPN have severe shortcomings in the feature extraction and fusion stages, such as making the extracted features lack of contextual and deep semantic information. In this work, we propose a non-local channel and spatial attention feature pyramid network (NCS-FPN) to improve multi-scale learning. In the feature extraction process, contextual semantic information from different scales is collected through non-local attention networks. In the feature fusion phase, deeper feature information from both spatial and context-aware sources is aggregated to enhance multi-scale feature extraction. Extensive experiments are carried out on two public datasets, MS COCO and PASCAL VOC, are carried out and the results demonstrate that NCS-FPN achieves better performance than several SOTA methods in most cases.

C2MA-Net: Cross-Modal Cross-Attention Network for Acute Ischemic Stroke Lesion Segmentation Based on CT Perfusion Scans.

Based on the hypothesis that adding a cross-modal and cross-attention (C2MA) mechanism into a deep learning network improves accuracy and efficacy of medical image segmentation, we propose to test a novel network to segment acute ischemic stroke (AIS) lesions from four CT perfusion (CTP) maps. The proposed network uses a C2MA module directly to establish a spatial-wise relationship by using the multigroup non-local attention operation between two modal features and performs dynamic group-wise recalibration through group attention block. This C2MA-Net has a multipath encoder-decoder architecture, in which each modality is processed in different streams on the encoding path, and the pair related parameter modalities are used to bridge attention across multimodal information through the C2MA module. A public dataset involving 94 training and 62 test cases are used to build and evaluate the C2MA-Net. AIS segmentation results on testing cases are analyzed and compared with other state-of-the-art models reported in the literature. By calculating several average evaluation scores, C2MA-network improves Recall and F2 scores by 6% and 1%, respectively. In the ablation experiment, the F1 score of C2MA-Net is at least 7.8% higher than that of single-input single-modal self-attention networks. This study demonstrates advantages of applying C2MA-network to segment AIS lesions, which yields promising segmentation accuracy, and achieves semantic decoupling by processing different parameter modalities separately. Proving the potential of cross-modal interactions in attention to assist identifying new imaging biomarkers for more accurately predicting AIS prognosis in future studies.

Non-local attention dual-branch network based cross-modal barefoot footprint retrieval

Non-local attention dual-branch network based cross-modal barefoot footprint retrieval

Utsn-Net: Medical Image Semantic Segmentation Model Based on Skip Non-Local Attention Module

Utsn-Net: Medical Image Semantic Segmentation Model Based on Skip Non-Local Attention Module

BANet: A Balance Attention Network for Anchor-Free Ship Detection in SAR Images

Recently, methods based on deep learning have been successfully applied to ship detection for synthetic aperture radar (SAR) images. However, most current ship detection networks rely too much on the anchor mechanism. These methods have low accuracy and poor generalization ability for multiscale ship detection. To solve the aforementioned problems, an anchor-free framework for multiscale ship detection in SAR images based on a balance attention network (BANet) is proposed. First, due to the diversity of scales and rotation angles of ships, deformable convolution is introduced to build a local attention module (LAM) to better obtain local information of ships and effectively enhance the robustness of the network. Second, a nonlocal attention module (NLAM) is designed to extract the nonlocal features of the SAR image, so as to balance the local features and nonlocal features acquired by the entire network. Finally, the feature pyramid network (FPN) is used to detect ships of different sizes at different scales. The detection results on three datasets demonstrate that the detection precision of our method is higher than that of all comparison methods, and this method achieves the most advanced performance.

M3Net: A multi-scale multi-view framework for multi-phase pancreas segmentation based on cross-phase non-local attention.

The complementation of arterial and venous phases visual information of CTs can help better distinguish the pancreas from its surrounding structures. However, the exploration of cross-phase contextual information is still under research in computer-aided pancreas segmentation. This paper presents M3Net, a framework that integrates multi-scale multi-view information for multi-phase pancreas segmentation. The core of M3Net is built upon a dual-path network in which individual branches are set up for two phases. Cross-phase interactive connections bridging the two branches are introduced to interleave and integrate dual-phase complementary visual information. Besides, we further devise two types of non-local attention modules to enhance the high-level feature representation across phases. First, we design a location attention module to generate cross-phase reliable feature correlations to suppress the misalignment regions. Second, the depth-wise attention module is used to capture the channel dependencies and then strengthen feature representations. The experiment data consists of 224 internal CTs (106 normal and 118 abnormal) with 1mm slice thickness, and 66 external CTs (29 normal and 37 abnormal) with 5mm slice thickness. We achieve new state-of-the-art performance with average DSC of 91.19% on internal data, and promising result with average DSC of 86.34% on external data.

Multilevel Progressive Network With Nonlocal Channel Attention for Hyperspectral Image Super-Resolution

Deep convolutional neural networks (CNNs) have made great progress in the super-resolution (SR) of hyperspectral images (HSIs). However, most methods utilize convolution to explore local features, and global features are ignored. It is expected that combining non-local mechanism with CNN will improve the performance of HSI SR. This paper presents a multi-level progressive HSI SR network. The dense non-local and local block (DNLB) is constructed to combine local and global features, which are used to reconstruct super-resolution images at each level. Due to the high dimension of HSI, original non-local methods produce memory-expensive attention maps. We develop a non-local channel attention block to extract the global features of HSIs efficiently. Spatial-spectral gradient is injected in the non-local attention block to obtain better details. Furthermore, the progressive learning mode based multi-level network is proposed to reconstruct HSI with fine details. A number of experiments demonstrate that our method can reconstruct hyperspectral images more accurately than existing methods.

Segmentation and Measurement of Superalloy Microstructure Based on Improved Nonlocal Block

The microstructure of superalloy materials has a decisive impact on its service performance. When preparing the material and photographing the microstructure, different depths of metallography perpendicular to the cut plane appear in the microstructure image. These metallographic features are major factors causing inaccurate segmentation. Aiming at the problems of traditional image processing methods, such as large noise and poor robustness of the edge extraction, the deep learning method is introduced. The receptive field of the traditional convolutional neural network method is too local to obtain remote dependencies, and dense feature extraction also brings problems such as excessive noise. To address the problem of non-tangential metallographic information in microstructure images that is not easily distinguishable. We consider how to combine and utilize the feature maps of the intermediate process as effectively as possible, and find that reusing affinity matrices from different angles and stacking them can improve the overall effect, and can solve the problem of inaccurate segmentation of metallographic images at different depths. We propose to improve and optimize the non-local attention module and further combine the module with the UNet network to form a new improved SNL-Unet image segmentation structure, which significantly raises the accuracy and efficiency of image segmentation, Additionally, we measure the characteristic parameters such as volume fraction, average thickness and the degree of rafting. The code for this paper will be available at github.com/ustbjdl1021/improved-snl-unet.

Variational Regularization Network With Attentive Deep Prior for Hyperspectral–Multispectral Image Fusion

Hyperspectral–multispectral image (HSI-MSI) fusion relies on a robust degradation model and data prior, where the former describes the degeneration of HSI in the spectral and spatial domains, and the latter reveals the latent statistics of the expected high-resolution (HR) HSI. In practice, the degradation model is often unknown, and the data prior is usually too complicated to be expressed analytically. In this study, we propose a variational network for HSI-MSI fusion (VaFuNet), in which the degradation model and data prior are implicitly represented by a deep learning network and jointly learned from the training data. A variational fusion model regularized by deep prior is first proposed, and then, it is optimized via a half-quadratic splitting and unfolded into a deep network. The deep prior is implicitly represented by a proximity operator. Due to the structural self-similarity, HSI possesses structural recurrences across different scales. To exploit such nonlocal prior and enhance the representability of network, we also propose a multiscale nonlocal attention and embed it into the deep prior proximity. The degradation model and deep prior proximity are jointly learned via end-to-end training. Experimental results on simulated and real-life HSI datasets demonstrate the effectiveness of the proposed VaFuNet HSI-MSI fusion method.

Learning Enriched Features for Image Super Resolution

In recent years, significant progress has been made in image super-resolution (SR) methods based on convolutional neural networks. However, most of them do not fully utilize multi-scale feature correspondence in the image SR process, resulting in blurred and artifact detail restoration, especially for SR tasks with larger scaling factors (i.e. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 4$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 8$ </tex-math></inline-formula> ). A multi-scale feature-enhanced SR network (MFENet) is proposed to solve the problems mentioned above. Specifically, the super-resolution method is enhanced by utilizing multi-scale feature correspondence in two ways. On the one hand, we propose an effective multi-scale non-local attention module and it can fully exploit a large number of low-level features at different scales of a single low-resolution (LR) feature map by exhaustively evaluating the correlations in local feature, in-scale non-local feature, and cross-scale non-local feature. On the other hand, the mapping from high-resolution (HR) to LR images is explored by a feedback branch we introduced, that provides an additional constraint for images reconstruction. It can penalize incorrect SR predictions to achieve better SR performance. Experimental results on 5 benchmark datasets show that our method outperforms state-of-the-art methods in terms of both subjective visual quality and objective quantitative metrics.

Res2Fusion: Infrared and Visible Image Fusion Based on Dense Res2net and Double Nonlocal Attention Models

Infrared and visible image fusion intends to generate a synthetic image with superior scene representation and better visual perception. The existing deep learning-based fusion methods merely make use of the convolution operation to extract features with a local receptive field, without fully considering their multiscale and long-range dependency characteristics, which may fail to preserve some essential global context from source images. To this end, we develop a novel and efficient fusion network based on dense Res2net and double nonlocal attention models, termed Res2Fusion. We introduce Res2net and dense connections into the encoder network with multiple available receptive fields, which is used to extract the multiscale features, and can retain as much meaningful information as possible for fusion tasks. In addition, we develop double nonlocal attention models as a fusion layer to model long-range dependencies on the local features. Specifically, these attention models can refine feature maps obtained by the encoder network to more focus on prominent infrared targets and distinct visible details. Finally, the comprehensive attention maps are used to generate a fused result through the simple decoder network. Extensive experiments demonstrate that the proposed method can simultaneously retain highlighted infrared targets and rich visible details and transcends other state-of-the-art fusion methods in terms of subjective and objective evaluation. The corresponding code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Zhishe-Wang/Res2Fusion</uri> .

An improved image enhancement framework based on multiple attention mechanism

Image enhancement can accentuate image feature and is necessary process in image processing. This work focuses on fusing multi-exposure image sequences low-light image enhancement. Inspired by the classical non-local means in computer vision, we proposed an improved deep neural network framework with attentions for image enhancement. Firstly, the original image was preprocessed in different dimensions. we get the edge images using an edge extracted algorithm and fusion multi exposed images to get an better initial images based on fully convolutional neural network with position and channel attention mechanism. Secondly, the head network is constructed by fully convolutional neural network. For capturing long-range dependencies between features maps, we designed a non-local attention module for head network to get better enhancement image. Finally, emerging the original images, edge image and fusion image as the input of the head network, it can enhance the images to get high-quality images. Experiments show that our framework proposed in this paper is effective and the attention mechanism play a significant hole in the network.

NL-VTON: a non-local virtual try-on network with feature preserving of body and clothes

In an image based virtual try-on network, both features of the target clothes and the input human body should be preserved. However, current techniques failed to solve the problems of blurriness on complex clothes details and artifacts on human body occlusion regions at the same time. To tackle this issue, we propose a non-local virtual try-on network NL-VTON. Considering that convolution is a local operation and limited by its convolution kernel size and rectangular receptive field, which is unsuitable for large size non-rigid transformations of persons and clothes in virtual try-on, we introduce a non-local feature attention module and a grid regularization loss so as to capture detailed features of complex clothes, and design a human body segmentation prediction network to further alleviate the artifacts on occlusion regions. The quantitative and qualitative experiments based on the Zalando dataset demonstrate that our proposed method significantly improves the ability to preserve features of bodies and clothes compared with the state-of-the-art methods.

DLNet: Accurate segmentation of green fruit in obscured environments

To achieve more accurate recognition and segmentation of obscured fruit in natural orchard environments, DLNet model is proposed. The model is improved for the more challenging problem of segmenting overlapping fruit from homochromatic backgrounds without considering various damages. This approach is tantamount to construct the detection network RS-RFP and the segmentation network DLNet. RS-RFP extends Full Convolutional One-Stage Object Detection (FCOS). Specifically, Feature Pyramid Network (FPN) by adding Gaussian non-local attention mechanism to build Refined Pyramid Network (RFP) for refining semantic features generated continuously by Residual Network (ResNet) and FPN. The DLNet segmentation framework is composed of a dual-layer Graph Attention Networks (GAT) layer is constructed to model the image as two overlapping layers, where the top GAT layer detects the occluded object (occluded) and the bottom GAT layer infers the partially occluded instance (occlude). Display modeling of the two-layer structure occlusion relationship can naturally the boundaries between the occluded and occlude instances and consider their interactions. The experimental results show that the method outperforms earlier segmentation models and achieves metric values of 80.9% and 81.2% for Average Precision (AP) box and AP mask respectively. In a reasonable running time, it meets the requirements of accuracy and robustness for picking robots and provides a reference for segmentation of other fruits and vegetables.

Single Image Super-Resolution Network Based on Multiple- Scale Self-Similarity Module

Recently, single image super-resolution (SISR) based on deep convolutional neural networks (CNNs) has been widely studied and achieved remarkable results. However, most of these methods mainly widen or deepen the network to get better results, ignoring the self-similarity in natural images. Some works have successfully leveraged this self-similarity by exploring non-local attention modules, but they only use non-local attention modules at one scale or two scales. And they need to calculate the self-similarity in each residual module, which makes the network more complex and takes a long time to train. We propose a multiple-scale self-similarity module (MSSS) that can be flexibly inserted into existing various super-resolution networks. MSSS uses convolution with different kernels and the trailing dilated convolution to obtain feature maps at different scales, and then calculates the correlation between multiple feature maps at different scales through two-head region-level recurrent criss-cross attention modules (TRRCCA), finally gets fusion output through channel attention. So MSSS can mine multiple-scale self-similarities. We only insert a MSSS module into the basic residual SISR network, and the result is significantly improved.

RS-Net: robust segmentation of green overlapped apples

Fruit detection and segmentation will be essential for future agronomic management, with applications in yield estimation, growth monitoring, intelligent picking, disease detection and etc. In order to more accurately and efficiently realize the recognition and segmentation of apples in natural orchards, a robust segmentation net framework specially developed for fruit production is proposed. This model was improved for the more challenging problem which segments the overlapped apples from the monochromatic background regardless of various corruptions. The method extends Mask R-CNN by embedding an attention mechanism for focusing more on the informative pixels but also suppressing the noise caused by adverse factors (occlusions, overlaps, etc.), which could be more suitable and robust for operating in complex natural environment. Specifically, the Gaussian non-local attention mechanism is transplanted into Mask R-CNN for refining the semantic features generated continuously by residual network and feature pyramid network, then the model forward processing based on the balanced feature levels and finally segments the regions where the apples are located. Experimental results verify the hypothesis of current work and show that the proposed method outperforms other start-of-the-art detection and segmentation models, the AP box and AP mask metric values have reached 85.6% and 86.2% in a reasonable run time, respectively, which can meet the precision and robustness of vision system in agronomic management.

Area-based correlation and non-local attention network for stereo matching

Area-based correlation and non-local attention network for stereo matching

Attention-Based 3D Human Pose Sequence Refinement Network.

Three-dimensional human mesh reconstruction from a single video has made much progress in recent years due to the advances in deep learning. However, previous methods still often reconstruct temporally noisy pose and mesh sequences given in-the-wild video data. To address this problem, we propose a human pose refinement network (HPR-Net) based on a non-local attention mechanism. The pipeline of the proposed framework consists of a weight-regression module, a weighted-averaging module, and a skinned multi-person linear (SMPL) module. First, the weight-regression module creates pose affinity weights from a 3D human pose sequence represented in a unit quaternion form. Next, the weighted-averaging module generates a refined 3D pose sequence by performing temporal weighted averaging using the generated affinity weights. Finally, the refined pose sequence is converted into a human mesh sequence using the SMPL module. HPR-Net is a simple but effective post-processing network that can substantially improve the accuracy and temporal smoothness of 3D human mesh sequences obtained from an input video by existing human mesh reconstruction methods. Our experiments show that the noisy results of the existing methods are consistently improved using the proposed method on various real datasets. Notably, our proposed method reduces the pose and acceleration errors of VIBE, the existing state-of-the-art human mesh reconstruction method, by 1.4% and 66.5%, respectively, on the 3DPW dataset.