Non-local Attention Research Articles

Revealing the Dark Side of Non-Local Attention in Single Image Super-Resolution.

Single Image Super-Resolution (SISR) aims to reconstruct a high-resolution image from its corresponding low-resolution input. A common technique to enhance the reconstruction quality is Non-Local Attention (NLA), which leverages self-similar texture patterns in images. However, we have made a novel finding that challenges the prevailing wisdom. Our research reveals that NLA can be detrimental to SISR and even produce severely distorted textures. For example, when dealing with severely degrade textures, NLA may generate unrealistic results due to the inconsistency of non-local texture patterns. This problem is overlooked by existing works, which only measure the average reconstruction quality of the whole image, without considering the potential risks of using NLA. To address this issue, we propose a new perspective for evaluating the reconstruction quality of NLA, by focusing on the sub-pixel level that matches the pixel-wise fusion manner of NLA. From this perspective, we provide the approximate reconstruction performance upper bound of NLA, which guides us to design a concise yet effective Texture-Fidelity Strategy (TFS) to mitigate the degradation caused by NLA. Moreover, the proposed TFS can be conveniently integrated into existing NLA-based SISR models as a general building block. Based on the TFS, we develop a Deep Texture-Fidelity Network (DTFN), which achieves state-of-the-art performance for SISR. Our code and a pre-trained DTFN are available on GitHub† for verification.

Application of deep learning for automatic detection of table tennis balls from an intelligent serving machine

The Intelligent Table Tennis Serving Machine (TTSM) has revolutionized table tennis training by simulating the variability and precision of human opponents. However, current systems lack the capability to track and analyze ball landing points, limiting comprehensive performance evaluation. To address this, we propose a novel approach utilizing YOLOv8 as the benchmark model for detecting table tennis balls launched by a TTSM, tailored to meet the demands of intelligent training systems. Our key innovation involves integrating a Res2Net architecture enhanced with a Non-local Attention Module (NLAM) and a Dilated Atrous Spatial Pyramid Pooling (DASPP) unit into the neck network. This design improves the network’s ability to integrate multi-scale local features, thereby enhancing global information processing. The DASPP module, with adaptive dilation convolution, ensures thorough context comprehension. Additionally, we incorporate an Omni-dimensional Dynamic Convolution (ODConv) module in the detection head, employing parallel strategies to learn diverse and complementary attention cues, further enhancing detection accuracy and robustness. To validate our method, we conducted evaluations using the publicly available OpenTTGames dataset and a self-constructed dataset comprising over 10,000 table tennis ball images captured in various environments. Experimental results indicate that our method enhances mAP0.5, Precision, and Recall by 2.3%, 5.1%, and 0.9% on the OpenTTGames dataset, and by 3%, 2%, and 1.6% on the self-built dataset, respectively. Finally, we describe the optimization process of the intelligent TTSM system, which leverages keyframe analysis of ball landing points to provide feedback, thereby enabling continuous performance improvement in subsequent phases.

Learnable dual attention fusion network for borehole distributed acoustic sensing systems data reconstruction

As the demand for nonrenewable oil and gas resources has increased, distributed acoustic sensing systems (DAS) have been widely used in the acquisition of vertical seismic profiles. Complicated subsurface conditions produce many types of noise with strong energy. The effective reflection and refraction signals are absorbed by the inelastic medium during the transmission process, resulting in significant energy loss and aliased frequency characteristics. However, satisfactory processing results and quality cannot be achieved through conventional denoising methods. In recent years, the rapid development of deep-learning methods with attention mechanisms has attracted significant attention. Noise reduction and signal reconstruction technologies with multiple attention mechanisms within the framework of joint-domain features have become important development directions in DAS data processing. An attention mechanism can help the network ignore useless information and pay attention to useful information in the same way that humans can. In this study, we discuss a novel learnable dual attention fusion network that combines local and patch-based nonlocal attention mechanisms based on ResBlock modules, making the extraction process of the network learnable and selective. In addition, the novel network can focus on local and global details. We also use a residual learning strategy to simplify the training process and gradually establish the best mapping based on the L2 loss function. All experimental results of the simulation and actual records show that our novel network has better denoising and effective signal restoration abilities. Moreover, it can recover the up-going and down-going wavefields and attenuate different types of noise with less effective energy loss.

Image Inpainting via Correlated Multi-Resolution Feature Projection.

With the advancement in image editing applications, image inpainting is gaining more attention due to its ability to recover corrupted images efficiently. Also, the existing methods for image inpainting either use two-stage coarse-to-fine architectures or single-stage architectures with a deeper network. On the other hand, shallow network architectures lack the quality of results and the methods with remarkable inpainting quality have high complexity in terms of number of parameters or average run time. Despite the improvement in the inpainting quality, these methods still lack the correlated local and global information. In this work, we propose a single-stage multi-resolution generator architecture for image inpainting with moderate complexity and superior outcomes. Here, a multi-kernel non-local (MKNL) attention block is proposed to merge the feature maps from all the resolutions. Further, a feature projection block is proposed to project features of MKNL to respective decoder for effective reconstruction of image. Also, a valid feature fusion block is proposed to merge encoder skip connection features at valid region and respective decoder features at hole region. This ensures that there will not be any redundant feature merging while reconstruction of image. Effectiveness of the proposed architecture is verified on CelebA-HQ Liu, et al. 2015, Karras et al. 2017, and Places2 Zhou et al. 2018 datasets corrupted with publicly available NVIDIA mask dataset Liu et al. 2018. The detailed ablation study, extensive result analysis, and application of object removal prove the robustness of the proposed method over existing state-of-the-art methods for image inpainting.

SS-CCDN: A semi-supervised pixel-wise concrete crack detection network using multi-task learning and memory information

Concrete cracks pose significant challenges to infrastructure maintenance and safety. Traditional methods for detecting cracks suffer from inefficiency and subjectivity. Deep learning has shown promise recently, yet it often requires extensive labeled data. A semi-supervised concrete crack detection network (SS-CCDN) is proposed to handle the aforementioned challenge. To be specific, a multi-task model that takes edge feature detection of cracks as an auxiliary task is established. This model is then applied to both student and teacher networks to realize semi-supervised detection. Furthermore, the feature fusion networks incorporate the innovative multi-head cosine non-local attention module to achieve comprehensive acquisition and implementation of feature information at different scales. Lastly, the memory module is integrated into the network to compare the commonalities and distinctions between input sample information and memory sample data, successfully and effectively detecting concrete crack areas. SS-CCDN outperforms other baselines in detection, according to experimental results on two benchmark datasets.

Multi-Stage Frequency Attention Network for Progressive Optical Remote Sensing Cloud Removal

Cloud contamination significantly impairs optical remote sensing images (RSIs), reducing their utility for Earth observation. The traditional cloud removal techniques, often reliant on deep learning, generally aim for holistic image reconstruction, which may inadvertently alter the intrinsic qualities of cloud-free areas, leading to image distortions. To address this issue, we propose a multi-stage frequency attention network (MFCRNet), a progressive paradigm for optical RSI cloud removal. MFCRNet hierarchically deploys frequency cloud removal modules (FCRMs) to refine the cloud edges while preserving the original characteristics of the non-cloud regions in the frequency domain. Specifically, the FCRM begins with a frequency attention block (FAB) that transforms the features into the frequency domain, enhancing the differentiation between cloud-covered and cloud-free regions. Moreover, a non-local attention block (NAB) is employed to augment and disseminate contextual information effectively. Furthermore, we introduce a collaborative loss function that amalgamates semantic, boundary, and frequency-domain information. The experimental results on the RICE1, RICE2, and T-Cloud datasets demonstrate that MFCRNet surpasses the contemporary models, achieving superior performance in terms of mean absolute error (MAE), root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM), validating its efficacy regarding the cloud removal from optical RSIs.

Emotional Inference from Speech Signals Informed by Multiple Stream DNNs Based Non-Local Attention Mechanism

Emotional Inference from Speech Signals Informed by Multiple Stream DNNs Based Non-Local Attention Mechanism

A cross-fusion of non-local attention network for infrared small target tracking

Infrared small target tracking plays important roles in many applications, such as infrared surveillance and reconnaissance. However, achieving promising performance with existing trackers is difficult due to the following challenges: (1) Infrared small targets lack color and texture information, making it difficult for the network to capture effective features; (2) A slight centroid-shift of the target can lead to tracking failure. To overcome these challenges, this paper proposes the CFNLA network, which consists of two parts: (1) The cross-fusion module is designed to capture the relationship between the target and its surroundings, including a non-local block to enhance the feature information of the target at the current scale and reduce interference such as background; (2) The confidence loss function is proposed to improve tracking stability by forcing the network to concentrate on centroid shift. Furthermore, a self-designed infrared small target tracking dataset, IRSTT, is constructed to evaluate the proposed algorithm. Finally, the experimental results on various open-source datasets and the IRSTT dataset verify the superior performance of the CFNLA network.

YOLOv8-QR: An improved YOLOv8 model via attention mechanism for object detection of QR code defects

Defect detection in Quick Response (QR) codes has important implications for downstream tasks. However, QR code defects include small objects and complex backgrounds, which makes their recognition effect poor. To address the above problems, we proposed a model based on YOLOv8, called YOLOv8-QR, to detect QR code defects. Specifically, first, the non-local attention (Non-local) module is introduced into the backbone of the YOLOv8 to enhance the interactive ability of the feature. The Non-local captures the correlation information of long-distance dependencies between features by calculating the attention weight between any positions. In addition, the contextual information required for representation learning of different defective objects is different. To extract multi-scale features, a Large Selective Kernel Network (LSKNet) was introduced. LSKNet dynamically adjusts the convolution receptive field of the neck fusion network and effectively uses the receptive field to capture the background information of different objects, thereby improving the representation ability of the model. To improve the defect detection accuracy of small objects in QR codes, the Normalized Gaussian Wasserstein Distance (NGWD) is introduced to replace the Intersection over Union (IoU) optimization function that is sensitive to the position deviation of objects and is not conducive to the regression of multi-scale objects. To verify the effectiveness of the model, the QR dataset was constructed and a series of experiments were conducted based on this dataset. The results show that the mAP50 and mAP50:95 of the YOLOv8-QR reach 95.5% and 65%, which are 3.8% and 2.3% higher than YOLOv8 respectively. The proposed YOLOv8-QR can better adapt to the needs of QR code defect detection in actual industrial environments. Our code is available at https://github.com/Code-of-Liujie/YOLOv8-QR.git.

The Geography of Investor Attention

Abstract Local companies attract significantly more attention from investors than nonlocal companies, especially at times of news releases and high volatility. This attention gap widens especially when news is firm-specific rather than aggregate, and in response to idiosyncratic rather than market risk. Attention is causally related to perceived proximity: after a firm is acquired by a nonlocal one, local investors, compared with nonlocal investors, are more likely to reallocate attention away from it; conversely, COVID-19 travel restrictions led investors to reallocate attention toward local companies and away from nonlocal ones, especially those difficult to reach. Finally, local attention predicts volatility, bid-ask spreads, and nonlocal attention, but not vice versa. Our findings suggest that the geography of attention matters and is shaped by local investors’ information-processing advantage, not familiarity bias. (JEL D83, G11, G12, G14, L86, R32) Received: 2 January 2024; Editorial decision: 13 March 2024 Editor: Isil Erel Authors have furnished an Internet Appendix, which is available on the Oxford University Press Web site next to the link to the final published paper online.

Stacked deformable convolution network with weighted non-local attention and branch residual connection for image quality assessment

Convolutional neural networks are data-driven Image Quality Assessment (IQA) models based on the convolutional operation in a rectangular window. Deformable convolutions with learnable receptive fields can efficiently extract structural features of irregular objects in an image. By superimposing deformable convolutions, a plug-and-play module is designed to obtain information for irregular geometric shapes. To selectively fuse shallow and deep features, we propose a weighted non-local attention (WNLA) module with the input and output of self-attention in a weighted manner. This paper proposes a dual branch residual full-reference IQA network that combines weighted non-local attention and stacked deformable convolution. The proposed network was trained on PIPAL dataset and tested on LIVE and TID2013. The cross-dataset evaluation shows that the network has a competitive generalization ability. Ablation experiments indicate that the proposed modules can effectively improve the performance of the network. Comparative experiments reveal that our network is superior to existing excellent networks. The codes for training, test and visualization are available at: https://github.com/Pengchang-haha/SDCN.git.

Field cabbage detection and positioning system based on improved YOLOv8n

BackgroundPesticide efficacy directly affects crop yield and quality, making targeted spraying a more environmentally friendly and effective method of pesticide application. Common targeted cabbage spraying methods often involve object detection networks. However, complex natural and lighting conditions pose challenges in the accurate detection and positioning of cabbage.ResultsIn this study, a cabbage detection algorithm based on the YOLOv8n neural network (YOLOv8-cabbage) combined with a positioning system constructed using a Realsense depth camera is proposed. Initially, four of the currently available high-performance object detection models were compared, and YOLOv8n was selected as the transfer learning model for field cabbage detection. Data augmentation and expansion methods were applied to extensively train the model, a large kernel convolution method was proposed to improve the bottleneck section, the Swin transformer module was combined with the convolutional neural network (CNN) to expand the perceptual field of feature extraction and improve edge detection effectiveness, and a nonlocal attention mechanism was added to enhance feature extraction. Ablation experiments were conducted on the same dataset under the same experimental conditions, and the improved model increased the mean average precision (mAP) from 88.8% to 93.9%. Subsequently, depth maps and colour maps were aligned pixelwise to obtain the three-dimensional coordinates of the cabbages via coordinate system conversion. The positioning error of the three-dimensional coordinate cabbage identification and positioning system was (11.2 mm, 10.225 mm, 25.3 mm), which meets the usage requirements.ConclusionsWe have achieved accurate cabbage positioning. The object detection system proposed here can detect cabbage in real time in complex field environments, providing technical support for targeted spraying applications and positioning.

SCGAFusion: A skip-connecting group convolutional attention network for infrared and visible image fusion

The goal of infrared and visible image fusion is to synthesize an image that contains complementary information as much as possible in paired infrared and visible images. However, due to using the convolution operation with local receptive field to extract features, most of the existing deep learning-based fusion methods fail to effectively exploit global contextual information in source images and so obtain a limited fusion performance. To address this issue, this paper proposes an end-to-end skip-connecting group convolutional attention network, termed as SCGAFusion, to fuse infrared and visible images. In SCGAFusion, a group convolutional attention block (GAB) is purposefully developed to promote the extraction and utilization capability of the fusion network for informative features. GAB introduces group convolutional layer to obtain hierarchical features, and employs a residual non-local attention module to capture long-range dependencies between pixels. In this way, it can not only focus more on the important regions or details, but also leverage both local neighborhood and global contextual information. Additionally, a hierarchical feature compensation mechanism based on skip connections is devised to integratedly exploit both local and global features. Experimental results on several public datasets qualitatively and quantitatively demonstrate the advantage of SCGAFusion over other state-of-the-art fusion methods.

Memory Augmentation and Non-Local Spectral Attention for Hyperspectral Denoising

In this paper, a novel hyperspectral denoising method is proposed, aiming at restoring clean images from images disturbed by complex noise. Previous denoising methods have mostly focused on exploring the spatial and spectral correlations of hyperspectral data. The performances of these methods are often limited by the effective information of the neighboring bands of the image patches in the spectral dimension, as the neighboring bands often suffer from similar noise interference. On the contrary, this study designed a cross-band non-local attention module with the aim of finding the optimal similar band for the input band. To avoid being limited to neighboring bands, this study also set up a memory library that can remember the detailed information of each input band during denoising training, fully learning the spectral information of the data. In addition, we use dense connected module to extract multi-scale spatial information from images separately. The proposed network is validated on both synthetic and real data. Compared with other recent hyperspectral denoising methods, the proposed method not only demonstrates good performance but also achieves better generalization.

Critical Information Mining Network: Identifying Crop Diseases in Noisy Environments

When agricultural experts explore the use of artificial intelligence technology to identify and detect crop diseases, they mainly focus on the research of a stable environment, but ignore the problem of noise in the process of image acquisition in real situations. To solve this problem, we propose an innovative solution called the Critical Information Mining Network (CIMNet). Compared with traditional models, CIMNet has higher recognition accuracy and wider application scenarios. The network has a good effect on crop disease recognition under noisy environments, and can effectively deal with the interference of noise to the recognition effect in actual farmland scenes. Consider that the shape of the leaves can be symmetrical or asymmetrical.First, we introduce the Non-Local Attention Module (Non-Local), which uses a unique self-attention mechanism to fully capture the context information of the image. The module overcomes the limitation of traditional convolutional neural networks that only rely on local features and ignore global features. Global features are particularly important when the image is disturbed by noise. Non-Local improves a more comprehensive visual understanding of crop disease recognition. Secondly, we have innovatively designed a Multi-scale Critical Information Fusion Module (MSCM). The module uses the Key Information Extraction Module (KIB) to dig into the shallow key features in the network deeply. The shallow key features strengthen the feature perception of the model to the noise image through texture and contour information, and then the shallow key features and deep features are fused to enrich the original deep feature information of the network. Finally, we conducted experiments on two public datasets, and the results showed that the accuracy of our model in crop disease identification under a noisy environment was significantly improved. At the same time, our model also showed excellent performance under stable conditions. The results of this study provide favorable support for the improvement of crop production efficiency.

Multimodal Fusion Network for 3-D Lane Detection.

3-D lane detection is a challenging task due to the diversity of lanes, occlusion, dazzle light, and so on. Traditional methods usually use highly specialized handcrafted features and carefully designed postprocessing to detect them. However, these methods are based on strong assumptions and single modal so that they are easily scalable and have poor performance. In this article, a multimodal fusion network (MFNet) is proposed through using multihead nonlocal attention and feature pyramid for 3-D lane detection. It includes three parts: multihead deformable transformation (MDT) module, multidirectional attention feature pyramid fusion (MA-FPF) module, and top-view lane prediction (TLP) ones. First, MDT is presented to learn and mine multimodal features from RGB images, depth maps, and point cloud data (PCD) for achieving optimal lane feature extraction. Then, MA-FPF is designed to fuse multiscale features for presenting the vanish of lane features as the network deepens. Finally, TLP is developed to estimate 3-D lanes and predict their position. Experimental results on the 3-D lane synthetic and ONCE-3DLanes datasets demonstrate that the performance of the proposed MFNet outperforms the state-of-the-art methods in both qualitative and quantitative analyses and visual comparisons.

IoT-Enhanced local attention dual networks for pathological image restoration in healthcare

High-resolution pathological images play a pivotal role in accurate disease diagnosis and are important in precision medicine. However, obtaining real-time high-resolution images faces challenges due to hardware limitations and scanning time constraints. Conventional image super-resolution restoration algorithms struggle to provide satisfactory results for pathological images, resulting in blurred and unrealistic restorations. In response to this challenge, this research proposes a pioneering approach by integrating Internet of Things (IoT) technology with Local Attention Dual Network (IOT-LAT) for super-resolution restoration of pathological images. The enhanced IOT-LATincorporates IoT-based data acquisition and processing, IoT-Enhanced non-local attention mechanisms, Gaussian constraint, and parameter-sharing strategies in up-sampling and down-sampling branches. This integration enables real-time super-resolution restoration of pathological images with improved accuracy. The reconstructed images exhibit a structural similarity of 0.914 and a peak signal-to-noise ratio of 30.84 dB. These results validate the effectiveness of the suggested approach in precisely reconstructing high-frequency details and improving modelling efficiency via a lightweight non-local attention mechanism enhanced by the Internet of Things. This work discusses IoT and non-local attention dual networks to improve pathological image super-resolution restoration. This allows for faster and more accurate disease diagnosis and treatment in precision medicine.

Enhancing unsupervised rock CT image super-resolution with non-local attention

Traditional low-resolution rock core images fail to capture the details and features of rocks, limiting the accurate analysis of rock properties. Therefore, researchers have proposed supervised and unsupervised digital rock super-resolution reconstruction methods. Supervised methods require paired data and manual annotation, while unsupervised methods generate high-resolution images through self-learning that are suitable for real-world scenarios with a lack of paired data. However, the current unsupervised methods still face challenges in terms of consistency and feature capturing. To address this problem, a novel unsupervised single-image super-resolution model called NL-CycleGAN is proposed. This model employs a non-local attention-guided CycleGAN framework to effectively capture low-level pixel variations between unpaired source and target images while preserving the overall tone. To evaluate the performance of NL-CycleGAN, we conduct both quantitative and qualitative tests using the DeepRock-SR dataset. In terms of quantitative evaluation, our method achieved the lowest perceptual loss (LPIPS) metric compared to existing methods. Additionally, our model obtained the highest performance in peak signal-to-noise ratio (PSNR), and multi-scale structural similarity index measure (MS-SSIM). In qualitative testing, the comparison images clearly show that our model accurately captures the intricate details of pores in carbonate and sandstone, as well as the complex crack patterns in coal. In conclusion, our model contributes to improving rock resolution and provides more accurate image processing techniques for rock property analysis and geological research.

Brain tumor segmentation using neuro-technology enabled intelligence-cascaded U-Net model.

According to experts in neurology, brain tumours pose a serious risk to human health. The clinical identification and treatment of brain tumours rely heavily on accurate segmentation. The varied sizes, forms, and locations of brain tumours make accurate automated segmentation a formidable obstacle in the field of neuroscience. U-Net, with its computational intelligence and concise design, has lately been the go-to model for fixing medical picture segmentation issues. Problems with restricted local receptive fields, lost spatial information, and inadequate contextual information are still plaguing artificial intelligence. A convolutional neural network (CNN) and a Mel-spectrogram are the basis of this cough recognition technique. First, we combine the voice in a variety of intricate settings and improve the audio data. After that, we preprocess the data to make sure its length is consistent and create a Mel-spectrogram out of it. A novel model for brain tumor segmentation (BTS), Intelligence Cascade U-Net (ICU-Net), is proposed to address these issues. It is built on dynamic convolution and uses a non-local attention mechanism. In order to reconstruct more detailed spatial information on brain tumours, the principal design is a two-stage cascade of 3DU-Net. The paper's objective is to identify the best learnable parameters that will maximize the likelihood of the data. After the network's ability to gather long-distance dependencies for AI, Expectation-Maximization is applied to the cascade network's lateral connections, enabling it to leverage contextual data more effectively. Lastly, to enhance the network's ability to capture local characteristics, dynamic convolutions with local adaptive capabilities are used in place of the cascade network's standard convolutions. We compared our results to those of other typical methods and ran extensive testing utilising the publicly available BraTS 2019/2020 datasets. The suggested method performs well on tasks involving BTS, according to the experimental data. The Dice scores for tumor core (TC), complete tumor, and enhanced tumor segmentation BraTS 2019/2020 validation sets are 0.897/0.903, 0.826/0.828, and 0.781/0.786, respectively, indicating high performance in BTS.

Adaptive selection of local and non-local attention mechanisms for speech enhancement

In speech enhancement tasks, local and non-local attention mechanisms have been significantly improved and well studied. However, a natural speech signal contains many dynamic and fast-changing acoustic features, and focusing on one type of attention mechanism (local or non-local) cannot precisely capture the most discriminative information for estimating target speech from background interference. To address this issue, we introduce an adaptive selection network to dynamically select an appropriate route that determines whether to use the attention mechanisms and which to use for the task. We train the adaptive selection network using reinforcement learning with a developed difficulty-adjusted reward that is related to the performance, complexity, and difficulty of target speech estimation from the noisy mixtures. Consequently, we propose an Attention Selection Speech Enhancement Network (ASSENet) with the innovative dynamic block that consists of an adaptive selection network and a local and non-local attention based speech enhancement network. In particular, the ASSENet incorporates both local and non-local attention and develops the attention mechanism selection technique to explore the appropriate route of local and non-local attention mechanisms for speech enhancement tasks. The results show that our method achieves comparable and superior performance to existing approaches with attractive computational costs.