Global Attention Module Research Articles

Scene Chinese Recognition with Local and Global Attention

Recently, scene Chinese recognition has attracted increasing attention. While mainstream scene text recognition methods exhibit outstanding performance in English recognition, they are considerably limited in Chinese recognition, due to inter-class similarity, intra-class variability, and complex combination of components in scene Chinese text. In this paper, we design Adaptive Position Encoding(APE) to enhance the model’s ability to perceive spatial information. Based on APE, we have innovatively designed Local Attention Module (LAM) and Global Attention Module (GAM). Specifically, LAM captures local features to identify common characteristics among characters of the same category, addressing the issue of intra-class variability. Meanwhile, LAM captures global features to identify the subordination relationships of Chinese character components. By integrating LAM and GAM, combining both local and global features, it is possible to find differences in the details among features that are fundamentally similar, thus solving the problem of inter-class similarity. Further, we contrive the transformer encoder–decoder structure to identify the vast variety of Chinese characters. Based on the Local/Global Attention Module and transformer encoder–decoder framework, we devise the novel sequence-to-sequence Local and Global Attention Network(LGANet), where both the backbone and the encoder/decoder are composed of attention mechanisms. Subsequent experiments on the Chinese scene dataset show that the recognition accuracy of our proposed LGANet is 77.3% and the normalized editing distance is 88.6%, both of which achieve the SOTA results in Fig. 1.

CRENet: Crowd region enhancement network for multi-person 3D pose estimation

Recovering multi-person 3D poses from a single image is a challenging problem due to inherent depth ambiguities, including root-relative depth and absolute root depth. Current bottom-up methods show promising potential to mitigate absolute root depth ambiguity through explicitly aggregating global contextual cues. However, these methods treat the entire image region equally during root depth regression, ignoring the negative impact of irrelevant regions. Moreover, they learn shared features for both depths, each of which focuses on different information. This sharing mechanism may result in negative transfer, thus diminishing root depth prediction accuracy. To address these challenges, we present a novel bottom-up method, Crowd Region Enhancement Network (CRENet), incorporating a Feature Decoupling Module (FDM) and a Global Attention Module (GAM). FDM explicitly learns the discriminative feature for each depth through adaptively recalibrating its channel-wise responses and fusing multi-level features, which makes the model focus on each depth prediction separately and thus avoids the adverse effect of negative transfer. GAM highlights crowd regions while suppressing irrelevant regions using the attention mechanism and further refines the attention regions based on the confidence measure about the attention, which is beneficial to learn depth-related cues from informative crowd regions and facilitate root depth estimation. Comprehensive experiments on benchmarks MuPoTS-3D and CMU Panoptic demonstrate that our method outperforms the state-of-the-art bottom-up methods in absolute 3D pose estimation and is applicable to in-the-wild images, which also indicates that learning depth-specific features and suppressing the noise signals can significantly benefit multi-person absolute 3D pose estimation.

SSM-Net: Semi-supervised multi-task network for joint lesion segmentation and classification from pancreatic EUS images

Pancreatic cancer does not show specific symptoms, which makes the diagnosis of early stages difficult with established image-based screening methods and therefore has the worst prognosis among all cancers. Although endoscopic ultrasonography (EUS) has a key role in diagnostic algorithms for pancreatic diseases, B-mode imaging of the pancreas can be affected by confounders such as chronic pancreatitis, which can make both pancreatic lesion segmentation and classification laborious and highly specialized. To address these challenges, this work proposes a semi-supervised multi-task network (SSM-Net) to leverage unlabeled and labeled EUS images for joint pancreatic lesion classification and segmentation. Specifically, we first devise a saliency-aware representation learning module (SRLM) on a large number of unlabeled images to train a feature extraction encoder network for labeled images by computing a contrastive loss with a semantic saliency map, which is obtained by our spectral residual module (SRM). Moreover, for labeled EUS images, we devise channel attention blocks (CABs) to refine the features extracted from the pre-trained encoder on unlabeled images for segmenting lesions, and then devise a merged global attention module (MGAM) and a feature similarity loss (FSL) for obtaining a lesion classification result. We collect a large-scale EUS-based pancreas image dataset (LS-EUSPI) consisting of 9,555 pathologically proven labeled EUS images (499 patients from four categories) and 15,500 unlabeled EUS images. Experimental results on the LS-EUSPI dataset and a public thyroid gland lesion dataset show that our SSM-Net clearly outperforms state-of-the-art methods.

A two-stream sign language recognition network based on keyframe extraction method

Sign language recognition (SLR) tasks are typically performed on a large number of continuous frames of sign language videos, making it challenging to utilize information efficiently and accurately from such a vast amount of data. Firstly, to efficiently utilize the information, it is necessary to select key information from numerous video frames to summarize the video. Therefore, this paper proposes a new method for key frame extraction that efficiently summarizes video information and addresses the issues of key frame clustering and data imbalance present in previous methods. Secondly, for more accurate information utilization, it is essential to learn the correct sign language expression features from a large dataset. Hence, we establish the Full-hand two-stream network to focus on the most crucial hand features in sign language expressions. The Full stream employs attention blocks to extract deep-level information and establishes temporal dependencies using global attention module. The Hand stream utilizes hand attention to focus on hand-specific feature information. Ultimately, our approach achieves state-of-the-art results on the CSL–500 Dataset and competitive results on the LSA64 Dataset. This paper validates the effectiveness of keyframe extraction methods and two-stream networks on the AUTSL dataset.

Mobile-YOLO: An accurate and efficient three-stage cascaded network for online fiberglass fabric defect detection

The tolerance for different defects in the production process of fiberglass fabric varies. For acceptable defects, only relevant information needs to be recorded and printed, while for intolerable defects, the loom needs to be stopped promptly for handling. To address the issue of inefficient production of fiberglass fabric due to this situation, this paper proposes a task-modularized cascade network model for quality inspection of fiberglass fabric. The three stages cascaded detection network is composed of the improved MobilenetV3-small and the enhanced You Only Look Once v8 Nano (YOLOv8n). For MobilenetV3_small, the Squeeze- and-Excitation(SE) module in the Block layer with a 5 × 5 convolutional kernel is replaced with the Efficient Channel Attention (ECA) mechanism, enhancing channel information while making the model more lightweight. For YOLOv8n, a Global Attention Module (GAM) is added to the network, enhancing the ability to capture global information of input data, making the model more perceptual. Additionally, a Similarity Attention Module (SimAM) is added between the neck and head networks, enabling the network to better capture relevant information in the images, improving the quality of feature maps and enhancing network performance. Finally, a large number of defect images of fiberglass fabric were collected from an industrial site, and different datasets for the three stages were created. Numerous experiments were conducted, and when using all three stages for detection, the accuracy rate reaches 99.4%, with a parameter count of 16.8M and Frames Per Second (FPS) of 186. The cascade network has been successfully applied in the industrial field of fiberglass fabric and meets industrial production requirements.

Deploying a Computer Vision Model Based on YOLOv8 Suitable for Drones in the Tuna Fishing and Aquaculture Industry

In recent years, the global tuna fishing and aquaculture industry has encountered significant challenges in balancing operational efficiency with sustainable resource management. This study introduces an innovative approach utilizing an advanced computer vision model, PA-YOLOv8, specifically adapted for drones, to enhance the monitoring and management of tuna populations. PA-YOLOv8 leverages the capabilities of YOLOv8, a state-of-the-art object detection system known for its precision and speed, tailored to address the unique demands of aerial surveillance in marine environments. Through comprehensive modifications including downsampling techniques, feature fusion enhancements, and the integration of the Global Attention Module (GAM), the model significantly improves the detection accuracy of small and juvenile tuna within complex aquatic landscapes. Experimental results using the Tuna dataset from Roboflow demonstrate marked improvements in detection metrics such as precision, recall, and mean average precision (mAP), affirming the model’s effectiveness. This study underscores the potential of integrating cutting-edge technologies like UAVs and computer vision in promoting sustainable practices in the aquaculture sector, setting a new standard for technological applications in environmental and resource management. The advancements presented here provide a scalable and efficient solution for real-time monitoring, contributing to the long-term sustainability of marine ecosystems.

Advancing cuffless blood pressure estimation: A PPG-based multi-task learning model for enhanced feature extraction and fusion

Cuffless continuous blood pressure (BP) monitoring is essential for personalized health management. Although existing cuffless BP estimation applies advanced machine learning techniques and integrates PPG signals, it is deficient in feature extraction and fusion. In addition, it is inefficient to train the model separately for different tasks. In this study, an advanced multi-domain and local–global feature parallel multi-task learning network (MDLG-MTLNet) is introduced. The MDLG-MTLNet was designed with three key aspects: first, temporal and multi-scale spatial features were extracted from PPG signals and their derivatives via a multi-scale spatial and temporal feature block; interaction of features from different domains was facilitated by the introduction of a local–global attention module that captured and efficiently fused local–global information; and lastly, intrinsic correlation between systolic (SBP) and diastolic blood pressure (DBP) was taken into account via a multi-task learning network to improve the overall performance of the model. On the MIMIC-II dataset, the MAEs of MDLG-MTLNet for DBP, SBP, and MBP were 2.64 mmHg, 1.57 mmHg, and 2.02 mmHg, respectively. These errors were superior to those of the existing methods, meeting the AAMI criteria, and earning an A grade according to the BHS protocol. The experimental results confirm the effectiveness of our proposed model in feature extraction and fusion.

Multi-robot social-aware cooperative planning in pedestrian environments using attention-based actor-critic

Safe and efficient cooperative planning of multiple robots in pedestrian participation environments is promising for applications. In this paper, a novel multi-robot social-aware efficient cooperative planner on the basis of off-policy multi-agent reinforcement learning (MARL) under partial dimension-varying observation and imperfect perception conditions is proposed. We adopt a temporal-spatial graph (TSG)-based social encoder to better extract the importance of social relations between each robot and the pedestrians in its field of view (FOV). Also, we introduce a K-step lookahead reward setting in the multi-robot RL framework to avoid aggressive, intrusive, short-sighted, and unnatural motion decisions generated by robots. Moreover, we improve the traditional centralized critic network with a multi-head global attention module to better aggregate local observation information among different robots to guide the process of the individual policy update. Finally, multi-group experimental results verify the effectiveness of the proposed cooperative motion planner.

Space dynamic target tracking method based on five-frame difference and Deepsort

For the problem of space dynamic target tracking with occlusion, this paper proposes an online tracking method based on the combination between the five-frame difference and Deepsort (Simple Online and Realtime Tracking with a Deep Association Metric), which is to achieve the identification first and then tracking of the dynamic target. First of all, according to three-frame difference, the five-frame difference is improved, and through the integration with ViBe (Visual Background Extraction), the accuracy and anti-interference ability are enhanced; Secondly, the YOLOv5s (You Look Only Once) is improved using preprocessing of DWT (Discrete Wavelet Transformation) and injecting GAM (Global Attention Module), which is considered as the detector for Deepsort to solve the missing in occlusion, and the real-time and accuracy can be strengthened; Lastly, simulation results show that the proposed space dynamic target tracking can keep stable to track all dynamic targets under the background interference and occlusion, the tracking precision is improved to 93.88%. Furthermore, there is a combination with the physical depth camera D435i, experiments on target dynamics show the effectiveness and superiority of the proposed recognition and tracking algorithm in the face of strong light and occlusion.

A Novel Boundary-Guided Global Feature Fusion Module for Instance Segmentation

The task of instance segmentation is widely acknowledged as being one of the most formidable challenges in the field of computer vision. Current methods have low utilization of boundary information, especially in dense scenes with occlusion and complex shapes of object instances, the boundary information may become ineffective. This results in coarse object boundary masks that fail to cover the entire object. To address this challenge, we are introducing a novel method called boundary-guided global feature fusion (BGF) which is based on the Mask R-CNN network. We designed a boundary branch that includes a Boundary Feature Extractor (BFE) module to extract object boundary features at different stages. Additionally, we constructed a binary image dataset containing instance boundaries for training the boundary branch. We also trained the boundary branch separately using a dedicated dataset before training the entire network. We then input the Mask R-CNN features and boundary features into a feature fusion module where the boundary features provide shape information needed for detection and segmentation. Finally, we use a global attention module (GAM) to further fuse features. Through extensive experiments, we demonstrate that our approach outperforms state-of-the-art instance segmentation algorithms, producing finer and more complete instance masks while also improving model capability.

TransVFS: A spatio-temporal local–global transformer for vision-based force sensing during ultrasound-guided prostate biopsy

Robot-assisted prostate biopsy is a new technology to diagnose prostate cancer, but its safety is influenced by the inability of robots to sense the tool-tissue interaction force accurately during biopsy. Recently, vision based force sensing (VFS) provides a potential solution to this issue by utilizing image sequences to infer the interaction force. However, the existing mainstream VFS methods cannot realize the accurate force sensing due to the adoption of convolutional or recurrent neural network to learn deformation from the optical images and some of these methods are not efficient especially when the recurrent convolutional operations are involved. This paper has presented a Transformer based VFS (TransVFS) method by leveraging ultrasound volume sequences acquired during prostate biopsy. The TransVFS method uses a spatio-temporal local–global Transformer to capture the local image details and the global dependency simultaneously to learn prostate deformations for force estimation. Distinctively, our method explores both the spatial and temporal attention mechanisms for image feature learning, thereby addressing the influence of the low ultrasound image resolution and the unclear prostate boundary on the accurate force estimation. Meanwhile, the two efficient local–global attention modules are introduced to reduce 4D spatio-temporal computation burden by utilizing the factorized spatio-temporal processing strategy, thereby facilitating the fast force estimation. Experiments on prostate phantom and beagle dogs show that our method significantly outperforms existing VFS methods and other spatio-temporal Transformer models. The TransVFS method surpasses the most competitive compared method ResNet3dGRU by providing the mean absolute errors of force estimation, i.e., 70.4 ± 60.0 millinewton (mN) vs 123.7 ± 95.6 mN, on the transabdominal ultrasound dataset of dogs.

Learned Focused Plenoptic Image Compression With Microimage Preprocessing and Global Attention

Focused plenoptic cameras can record spatial and angular information of the light field (LF) simultaneously with higher spatial resolution relative to traditional plenoptic cameras, which facilitate various applications in computer vision. However, the existing plenoptic image compression methods present ineffectiveness to the captured images due to the complex micro-textures generated by the microlens relay imaging and long-distance correlations among the microimages. In this paper, a lossy end-to-end learning architecture is proposed to compress the focused plenoptic images efficiently. First, a data preprocessing scheme is designed according to the imaging principle to remove the sub-aperture image ineffective pixels in the recorded light field and align the microimages to the rectangular grid. Then, the global attention module with large receptive field is proposed to capture the global correlation among the feature maps using pixel-wise vector attention computed in the resampling process. Also, a new image dataset consisting of 1910 focused plenoptic images with content and depth diversity is built to benefit training and testing. Extensive experimental evaluations demonstrate the effectiveness of the proposed approach. It outperforms intra coding of HEVC and VVC by an average of 62.57% and 51.67% bitrate reduction on the 20 preprocessed focused plenoptic images, respectively. Also, it achieves 18.73% bitrate saving and generates perceptually pleasant reconstructions compared to the state-of-the-art end-to-end image compression methods, which benefits the applications of focused plenoptic cameras greatly. The dataset and code are publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/VincentChandelier/GACN</uri> .

Self-Adaptive Graph With Nonlocal Attention Network for Skeleton-Based Action Recognition.

Graph convolutional networks (GCNs) have achieved encouraging progress in modeling human body skeletons as spatial-temporal graphs. However, existing methods still suffer from two inherent drawbacks. Firstly, these models process the input data based on the physical structure of the human body, which leads to some latent correlations among joints being ignored. Furthermore, the key temporal relationships between nonadjacent frames are overlooked, preventing to fully learn the changes of the body joints along the temporal dimension. To address these issues, we propose an innovative spatial-temporal model by introducing a self-adaptive GCN (SAGCN) with global attention network, collectively termed SAGGAN. Specifically, the SAGCN module is proposed to construct two additional dynamic topological graphs to learn the common characteristics of all data and represent a unique pattern for each sample, respectively. Meanwhile, the global attention module (spatial attention (SA) and temporal attention (TA) modules) is designed to extract the global connections between different joints in a single frame and model temporal relationships between adjacent and nonadjacent frames in temporal sequences. In this manner, our network can capture richer features of actions for accurate action recognition and overcome the defect of the standard graph convolution. Extensive experiments on three benchmark datasets (NTU-60, NTU-120, and Kinetics) have demonstrated the superiority of our proposed method.

High-Throughput Spike Detection in Greenhouse Cultivated Grain Crops with Attention Mechanisms-Based Deep Learning Models.

Detection of spikes is the first important step toward image-based quantitative assessment of crop yield. However, spikes of grain plants occupy only a tiny fraction of the image area and often emerge in the middle of the mass of plant leaves that exhibit similar colors to spike regions. Consequently, accurate detection of grain spikes renders, in general, a non-trivial task even for advanced, state-of-the-art deep neural networks (DNNs). To improve pattern detection in spikes, we propose architectural changes to Faster-RCNN (FRCNN) by reducing feature extraction layers and introducing a global attention module. The performance of our extended FRCNN-A vs. conventional FRCNN was compared on images of different European wheat cultivars, including "difficult" bushy phenotypes from 2 different phenotyping facilities and optical setups. Our experimental results show that introduced architectural adaptations in FRCNN-A helped to improve spike detection accuracy in inner regions. The mean average precision (mAP) of FRCNN and FRCNN-A on inner spikes is 76.0% and 81.0%, respectively, while on the state-of-the-art detection DNNs, Swin Transformer mAP is 83.0%. As a lightweight network, FRCNN-A is faster than FRCNN and Swin Transformer on both baseline and augmented training datasets. On the FastGAN augmented dataset, FRCNN achieved a mAP of 84.24%, FRCNN-A attained a mAP of 85.0%, and the Swin Transformer achieved a mAP of 89.45%. The increase in mAP of DNNs on the augmented datasets is proportional to the amount of the IPK original and augmented images. Overall, this study indicates a superior performance of attention mechanisms-based deep learning models in detecting small and subtle features of grain spikes.

Multi-level feature fusion network combining attention mechanisms for polyp segmentation

Clinically, automated polyp segmentation techniques have the potential to significantly improve the efficiency and accuracy of medical diagnosis, thereby reducing the risk of colorectal cancer in patients. Unfortunately, existing methods suffer from two significant weaknesses that can impact the accuracy of segmentation. Firstly, features extracted by encoders are not adequately filtered and utilized. Secondly, semantic conflicts and information redundancy caused by feature fusion are not attended to. To overcome these limitations, we propose a novel approach for polyp segmentation, named MLFF-Net, which leverages multi-level feature fusion and attention mechanisms. Specifically, MLFF-Net comprises three modules: Multi-scale Attention Module (MAM), High-level Feature Enhancement Module (HFEM), and Global Attention Module (GAM). Among these, MAM is used to extract multi-scale information and polyp details from the shallow output of the encoder. In HFEM, the deep features of the encoders complement each other by aggregation. Meanwhile, the attention mechanism redistributes the weight of the aggregated features, weakening the conflicting redundant parts and highlighting the information useful to the task. GAM combines features from the encoder and decoder features, as well as computes global dependencies to prevent receptive field locality. Experimental results on five public datasets show that the proposed method not only can segment multiple types of polyps but also has advantages over current state-of-the-art methods in both accuracy and generalization ability.

Foreign-Object Detection in High-Voltage Transmission Line Based on Improved YOLOv8m

The safe operation of high-voltage transmission lines ensures the power grid’s security. Various foreign objects attached to the transmission lines, such as balloons, kites and nesting birds, can significantly affect the safe and stable operation of high-voltage transmission lines. With the advancement of computer vision technology, periodic automatic inspection of foreign objects is efficient and necessary. Existing detection methods have low accuracy because foreign objects attached to the transmission lines are complex, including occlusions, diverse object types, significant scale variations, and complex backgrounds. In response to the practical needs of the Yunnan Branch of China Southern Power Grid Co., Ltd., this paper proposes an improved YOLOv8m-based model for detecting foreign objects on transmission lines. Experiments are conducted on a dataset collected from Yunnan Power Grid. The proposed model enhances the original YOLOv8m by incorporating a Global Attention Module (GAM) into the backbone to focus on occluded foreign objects, replacing the SPPF module with the SPPCSPC module to augment the model’s multiscale feature extraction capability, and introducing the Focal-EIoU loss function to address the issue of high- and low-quality sample imbalances. These improvements accelerate model convergence and enhance detection accuracy. The experimental results demonstrate that our proposed model achieves a 2.7% increase in mAP_0.5, a 4% increase in mAP_0.5:0.95, and a 6% increase in recall.

MAFormer: A New Method for Radar Reflectivity Reconstructing Using Satellite Data

Radar reflectivity plays a crucial role in detecting heavy rainfall and is an important tool for meteorological analysis. However, the coverage of a single radar is limited, leading to the use of satellite data as a complementary source. Consequently, how to bridge the gap between radar and satellite data has become a growing research focus. In this paper, we present MAFormer, a novel model for reconstructing radar reflectivity using satellite data within the Transformer framework. MAFormer consists of two modules: the Axial Local Attention Module and the Mixup Global Attention Module, which extract both local saliency and global similarity. Quantitative and qualitative experiments demonstrate the effectiveness of our proposed method. Specifically, the MAFormer model exhibits notable advancements when compared to state-of-the-art deep learning techniques. It demonstrates an improvement ranging from 0.01 to 0.05 in terms of the Heidke skill score, indicating its superior performance. Additionally, MAFormer effectively mitigates false alarm rates by approximately 0.016 to 0.04, which further highlights its enhanced accuracy and reliability.

Ultrasound Image Classification of Thyroid Nodules Based on Attention Mechanism

Ultrasound imaging of the thyroid gland is currently the most common diagnostic technique, used for pre diagnosis of benign and malignant thyroid nodules. During the clinical diagnosis, the condition judgment requires the high clinical experience of physicians. The low level of medical care in remote areas has led to an increased rate of misdiagnosis of benign and malignant nodules. To improve the accuracy of diagnosis and treatment effectiveness of doctors, this work proposes a benign and malignant thyroid classification model based on mechanisms of attention. First, this paper performs data cleaning on the collected data and performs pre-processing operations such as data enhancement to construct a new dataset. Then, a network model based on Lightweight Global Attention Module (LGAM) is constructed for the benign and malignant classification of nodules. Finally, it is shown by experimental analysis and comparison that the improved LGAM-based thyroid node classification model has achieved better experimental results on the dataset and can achieve the optimal classification accuracy of 95.16% for the network model proposed in this work.

Automatic Matching of Optical and SAR Images Based on Attention Structure Features

Due to the disparity in imaging techniques, significant radiometric and geometric variances exist among optical and Synthetic Aperture Radar (SAR) images, making it a challenging task for achieving automatic and accurate matching in contemporary international academic research. Handcrafted structural features have shown some success in heterogeneous image matching in recent years. However, improving its matching performance manually proves to be difficult. As a result, this work presents a matching strategy based on attention-enhanced structural feature representation to improve optical and SAR images matching accuracy. In this research, a novel multi-branch global attention module is built using handmade structural feature extraction. This module can focus on the common information of structural feature descriptors in space and channel, extracting finer and more robust image features. Then, the proposed method utilizes the sum of squared difference (SSD) learning metric, which is based on the fast Fourier transform, to develop a loss function. This loss function is then used to train positive and negative samples in order to enhance the discriminative ability of the model. Experimental results obtained from training and testing on numerous optical and SAR datasets demonstrate that the proposed method significantly improves the accuracy of matching optical and SAR images compared to both current structural feature matching methods and advanced deep learning matching models.

3D pulmonary vessel segmentation based on improved residual attention u-net

Automatic segmentation of pulmonary vessels is a fundamental and essential task for the diagnosis of various pulmonary vessels diseases. The accuracy of segmentation is suffering from the complex vascular structure. In this paper, an Improved Residual Attention U-Net (IRAU-Net) aiming to segment pulmonary vessel in 3D is proposed. To extract more vessel structure information, the Squeeze and Excitation (SE) block is embedded in the down sampling stage. And in the up sampling stage, the global attention module (GAM) is used to capture target features in both high and low levels. These two stages are connected by Atrous Spatial Pyramid Pooling (ASPP) which can sample in various receptive fields with a low computational cost. By the evaluation experiment, the better performance of IRAU-Net on the segmentation of terminal vessel is indicated. It is expected to provide robust support for clinical diagnosis and treatment.