Spatial Channel Attention Research Articles

RMANet: Refined-mixed attention network for progressive low-light image enhancement

Multi-scale feature fusion has been recognized as an effective strategy to boost the quality of low-light images. However, most existing methods directly extract multi-scale contextual information from severely degraded and down-sampled low-light images, resulting in a large amount of unexpected noise and degradation contaminating the learned multi-scale features. Moreover, there exist large redundant and overlapping features when directly concatenating multi-scale feature maps, which fails to consider different contributions of different scales. To conquer the above challenges, this paper presents a novel approach termed progressive Refined-Mixed Attention Network (RMANet) for low-light image enhancement. The proposed RMANet first targets a single-scale pre-enhancement and then progressively increases multi-scale spatial-channel attention fusion in a coarse-to-fine fashion. Additionally, we elaborately devise a Refined-Mixed Attention Module (RMAM) to first learn a parallel spatial-channel dominant features and then selectively integrate dominant features in the spatial and channel dimensions across multiple scales. Noticeably, our proposed RMANet is a lightweight yet flexible end-to-end framework that adapts to diverse application scenarios. Thorough experiments carried out upon three popular benchmark databases demonstrate that our approach surpasses existing methods in terms of both quantitative quality metrics and visual quality assessment. The code will be available at https://github.com/kbzhang0505/RMANet.

Reconstruction residual network with a fused spatial-channel attention mechanism for automatically classifying diabetic foot ulcer.

Diabetic foot ulcer (DFU) is a common chronic complication of diabetes. This complication is characterized by the formation of ulcers that are difficult to heal on the skin of the foot. Ulcers can negatively affect patients' quality of life, and improperly treated lesions can result in amputation and even death. Traditionally, the severity and type of foot ulcers are determined by doctors through visual observations and on the basis of their clinical experience; however, this subjective evaluation can lead to misjudgments. In addition, quantitative methods have been developed for classifying and scoring are therefore time-consuming and labor-intensive. In this paper, we propose a reconstruction residual network with a fused spatial-channel attention mechanism (FARRNet) for automatically classifying DFUs. The use of pseudo-labeling and Data augmentation as a pre-processing technique can overcome problems caused by data imbalance and small sample size. The developed model's attention was enhanced using a spatial channel attention (SPCA) module that incorporates spatial and channel attention mechanisms. A reconstruction mechanism was incorporated into the developed residual network to improve its feature extraction ability for achieving better classification. The performance of the proposed model was compared with that of state-of-the-art models and those in the DFUC Grand Challenge. When applied to the DFUC Grand Challenge, the proposed method outperforms other state-of-the-art schemes in terms of accuracy, as evaluated using 5-fold cross-validation and the following metrics: macro-average F1-score, AUC, Recall, and Precision. FARRNet achieved the F1-score of 60.81%, AUC of 87.37%, Recall of 61.04%, and Precision of 61.56%. Therefore, the proposed model is more suitable for use in medical diagnosis environments with embedded devices and limited computing resources. The proposed model can assist patients in initial identifications of ulcer wounds, thereby helping them to obtain timely treatment.

Building detection in VHR remote sensing images using a novel dual attention residual-based U-Net (DAttResU-Net): An application to generating building change maps

In today's era, increasing access to very high-resolution remote sensing images (VHR-RSIs) has enhanced building detection and change assessment capabilities. These applications provide accurate urban mapping, facilitate effective land management, and support disaster assessment by delivering detailed insights into building structures and their temporal changes. This study uses a two-stage process to present a pioneering approach for generating precise building maps (BMs) and subsequent building change maps (BCMs) from VHR-RSIs. The primary question addressed by the research is how to enhance the U-Net architecture to improve its sensitivity to both high-level semantic features (HLSF) and low-level spatial features (LLSF) in the building detection task. For this purpose, in the initial stage of the method, a novel deep learning model called dual attention residual-based U-Net (DAttResU-Net) is introduced. This model incorporates two significant modifications to the conventional U-Net, enhancing its capacity to yield bi-temporal BMs. Firstly, each standard convolutional block (CB) is replaced with an optimized CB incorporating a channel-spatial attention module attuned to the building objects' crucial HLSF. Secondly, an additional attention module is integrated into the encoder-decoder path of the model, heightening the sensitivity of U-Net to vital LLSF of buildings while disregarding extraneous background spatial information during the fusion of HLSF and LLSF. In the subsequent stage, the bi-temporal BMs generated by the DAttResU-Net are subjected to a box-based class-object change detection methodology to produce accurate BCMs. The effectiveness of the proposed architecture is rigorously evaluated against state-of-the-art models in both BM and BCM generation contexts, utilizing the well-established WHU dataset for experimentation. The experimental results indicated that the DAttResU-Net model, boasting an average of PFN/ PFP value of 2.33/1.34 (%) surpasses the performance of the state-of-the-art models in generating bi-temporal BMs. Furthermore, the building change detection outcomes demonstrated the proficient role of the bi-temporal BMs predicted by the proposed model in leading to the most optimal BCMs, exhibiting average PFN/ PFP value of 2.63/8.93 (%), outperforming comparative networks. Finally, we concluded that the proposed DAttResU-Net architecture is a highly promising and applicable model for producing reliable BMs and BCMs.

A Lightweight Insulator Defect Detection Model Based on Drone Images

With the continuous development and construction of new power systems, using drones to inspect the condition of transmission line insulators has become an inevitable trend. To facilitate the deployment of drone hardware equipment, this paper proposes IDD-YOLO (Insulator Defect Detection-YOLO), a lightweight insulator defect detection model. Initially, the backbone network of IDD-YOLO employs GhostNet for feature extraction. However, due to the limited feature extraction capability of GhostNet, we designed a lightweight attention mechanism called LCSA (Lightweight Channel-Spatial Attention), which is combined with GhostNet to capture features more comprehensively. Secondly, the neck network of IDD-YOLO utilizes PANet for feature transformation and introduces GSConv and C3Ghost convolution modules to reduce redundant parameters and lighten the network. The head network employs the YOLO detection head, incorporating the EIOU loss function and Mish activation function to optimize the speed and accuracy of insulator defect detection. Finally, the model is optimized using TensorRT and deployed on the NVIDIA Jetson TX2 NX mobile platform to test the actual inference speed of the model. The experimental results demonstrate that the model exhibits outstanding performance on both the proprietary ID-2024 insulator defect dataset and the public SFID insulator dataset. After optimization with TensorRT, the actual inference speed of the IDD-YOLO model reached 20.83 frames per second (FPS), meeting the demands for accurate and real-time inspection of insulator defects by drones.

MSCS: Multi-stage feature learning with channel-spatial attention mechanism for infrared and visible image fusion

The intention of infrared and visible image fusion is to combine the images captured by different modal sensors in the same scene to enhance its understanding. Deep learning has been proven its powerful application in image fusion due to its fine generalization, robustness, and representability of deep features. However, the performance of these deep learning-based methods heavily depends on the illumination condition. Especially in dark or exposed scenes, the fused results are over-smoothness and low-contrast, resulting in inaccuracy of object detection. To address these issues, this paper develops a multi-stage feature learning approach with channel-spatial attention mechanism, namely MSCS, for infrared and visible image fusion. The MSCS is composed of the following four key procedures: Firstly, the infrared and visible images are decomposed into illumination and reflectance components by a proposed network called as Retinex_Net. Then, the components are transported to an encoder for features coding. Next, we propose an adaptive fusion module with attention mechanisms to fuse the features. Finally, the fused image is generated by the decoder for decoding the fused features. Meanwhile, a novel fusion loss function and a multi-stage training strategy are proposed to train the above modules. The subjective and objective results of experiments on TNO, LLVIP and MSRS datasets illustrate that the proposed method is effective and performs better than the state-of-the-art fusion methods on achieving enjoyable results in dark or over-exposure scenes. And the results of further experiments on the fused images for object detection demonstrate that the fusion outputs produced by our MSCS are more beneficial for detection tasks.

ICRBP-LKHA: Large convolutional kernel and hybrid channel-spatial attention for identifying circRNA-RBP interaction sites.

Circular RNAs (circRNAs) play vital roles in transcription and translation. Identification of circRNA-RBP (RNA-binding protein) interaction sites has become a fundamental step in molecular and cell biology. Deep learning (DL)-based methods have been proposed to predict circRNA-RBP interaction sites and achieved impressive identification performance. However, those methods cannot effectively capture long-distance dependencies, and cannot effectively utilize the interaction information of multiple features. To overcome those limitations, we propose a DL-based model iCRBP-LKHA using deep hybrid networks for identifying circRNA-RBP interaction sites. iCRBP-LKHA adopts five encoding schemes. Meanwhile, the neural network architecture, which consists of large kernel convolutional neural network (LKCNN), convolutional block attention module with one-dimensional convolution (CBAM-1D) and bidirectional gating recurrent unit (BiGRU), can explore local information, global context information and multiple features interaction information automatically. To verify the effectiveness of iCRBP-LKHA, we compared its performance with shallow learning algorithms on 37 circRNAs datasets and 37 circRNAs stringent datasets. And we compared its performance with state-of-the-art DL-based methods on 37 circRNAs datasets, 37 circRNAs stringent datasets and 31 linear RNAs datasets. The experimental results not only show that iCRBP-LKHA outperforms other competing methods, but also demonstrate the potential of this model in identifying other RNA-RBP interaction sites.

Insulator detection based on FA‐YOLO network with improved feature extraction ability

AbstractUnmanned aerial vehicle insulator detection that aims to recognize defective insulators from transmission lines has made significant progress in recent years. However, it still faces challenges, such as the complex background of aerial images and the small memory of unmanned aerial vehicles. This paper proposes a refined insulator detection algorithm that integrates the attention mechanism in YOLOv8 to improve the feature extraction ability. Specifically, this paper introduces a fast vision transformers structure in the you only look once (YOLO) v8 backbone section to enhance feature extraction by capturing local and global features. Additionally, the global attention mechanism is incorporated in the neck for additional feature extraction by merging comprehensive spatial and channel information into the output. Furthermore, we amalgamate depth‐wise convolution, graph convolution, and residual operation in the global attention mechanism module. This design can mitigate the issues of gradient vanishing or exploding and meanwhile enhance the distinction between spatial attention and channel attention. The proposed model is then applied to a public dataset and a set of real images from a specific power station, and the detection results show that it outperforms many competitors in terms of accuracy, efficiency, and memory size.

Channel Spatial Attention for automated detection of macular diseases

The performance of a classifier heavily depends on the specific set of features used for the classification task. Accurate classification is crucial in several fields, including engineering and medicine. We propose a feature attention-based technique for automatically recognising macular disorders from retinal Optical Coherence Tomography (OCT) images. By emphasising specific features within regions of interest, the attention function lessens the impact of irrelevant information. Deep convolutional neural networks (CNNs) often incorporate attention methods that combine channel and spatial knowledge to improve model performance. However, these approaches rely on 2D global pooling operations or scaling techniques, which can lead to information loss. We introduce a Channel Spatial Attention approach to capture cross-dimensional interactions, thereby mitigating information loss. This approach achieves significant performance improvements with minimal computational overhead and can seamlessly integrate into any CNN. We validated our proposed strategy using the UCSD benchmark dataset to affirm its significance. The study's results confirm the efficacy and usefulness of our approach in detecting macular diseases from retinal OCT images.

RS-Dseg: semantic segmentation of high-resolution remote sensing images based on a diffusion model component with unsupervised pretraining

Semantic segmentation plays a crucial role in interpreting remote sensing images, especially in high-resolution scenarios where finer object details, complex spatial information and texture structures exist. To address the challenge of better extracting semantic information and ad-dressing class imbalance in multiclass segmentation, we propose utilizing diffusion models for remote sensing image semantic segmentation, along with a lightweight classification module based on a spatial-channel attention mechanism. Our approach incorporates unsupervised pretrained components with a classification module to accelerate model convergence. The diffusion model component, built on the UNet architecture, effectively captures multiscale features with rich contextual and edge information from images. The lightweight classification module, which leverages spatial-channel attention, focuses more efficiently on spatial-channel regions with significant feature information. We evaluated our approach using three publicly available datasets: Postdam, GID, and Five Billion Pixels. In the test of three datasets, our method achieved the best results. On the GID dataset, the overall accuracy was 96.99%, the mean IoU was 92.17%, and the mean F1 score was 95.83%. In the training phase, our model achieved good performance after only 30 training cycles. Compared with other models, our method reduces the number of parameters, improves the training speed, and has obvious performance advantages.

Dual-branch collaborative Siamese network for visual tracking

This paper introduces a dual-branch collaborative Siamese network architecture, CoSiNet, for visual object tracking. The network has a shallow branch for precise target localization and a deep branch for extracting rich semantic information. It integrates two specialized modules - Channel Attention and Spatial Channel Attention Feature Enhancement - for improved feature extraction and background noise reduction. An Adaptive Fusion Module combines response maps from both branches to create an enriched final response map. Experimental results show our algorithm outperforms several contemporary techniques on four different public datasets (OTB100, OTB50, DTB70, TColor128), demonstrating the competitiveness of CoSiNet.

ALKU-Net: Adaptive Large Kernel Attention Convolution Network for Lung Nodule Segmentation

The accurate segmentation of lung nodules in computed tomography (CT) images is crucial for the early screening and diagnosis of lung cancer. However, the heterogeneity of lung nodules and their similarity to other lung tissue features make this task more challenging. By using large receptive fields from large convolutional kernels, convolutional neural networks (CNNs) can achieve higher segmentation accuracies with fewer parameters. However, due to the fixed size of the convolutional kernel, CNNs still struggle to extract multi-scale features for lung nodules of varying sizes. In this study, we propose a novel network to improve the segmentation accuracy of lung nodules. The network integrates adaptive large kernel attention (ALK) blocks, employing multiple convolutional layers with variously sized convolutional kernels and expansion rates to extract multi-scale features. A dynamic selection mechanism is also introduced to aggregate the multi-scale features obtained from variously sized convolutional kernels based on selection weights. Based on this, we propose a lightweight convolutional neural network with large convolutional kernels, called ALKU-Net, which integrates the ALKA module in a hierarchical encoder and adopts a U-shaped decoder to form a novel architecture. ALKU-Net efficiently utilizes the multi-scale large receptive field and enhances the model perception capability through spatial attention and channel attention. Extensive experiments demonstrate that our method outperforms other state-of-the-art models on the public dataset LUNA-16, exhibiting considerable accuracy in the lung nodule segmentation task.

Mapping brain development against neurological disorder using contrastive sharing

Cerebral palsy (CP) is a neurological disorder caused by cerebral ischemia and hypoxia during fetal brain development. Early intervention in CP patients has a favorable impact on medications and therapy; however, challenges are associated with early age development, potential recovery, and neuroimaging sensitivity. Previously, the main focus was given on early diagnosis, brain injury characterization, and neurodevelopmental outcomes in infants; however, it is inevitable to study the brain developmental patterns in health controls (HC) and CP children. This study elaborates on the effects of aging on the brains of infants with HC and CP, age ranging from a few months and 17 years using deep learning (DL). The introduced DL-based models, namely SCT-DL and MCT-DL, trained on single contrast MRI (SC-MRI) and coupling of multi-contrast (MC-MRI) to learn vulnerable brain regions associated to brain development. The proposed models are equipped with fully-convolutional-based attention mechanism leveraging average pooling operation, directly passing salient features, parallel partial computing unit, specialized parameter sharing module, fusion, and spatial-channel attention with traversal placement to learn and associate brain development accurately. In addition, the study reports optimal SC-MRI and MC-MRI deeply associated with CP vulnerabilities and association to brain development. The SCT-DL outperformed the counterpart models with accumulative MAE = 1.73 (C11C denotes HC and P denotes CP.= 1.61 and P = 1.63) underlying T1-w, where the MCT-DL is the improved version of SCT-DL and the prediction accuracy reached to MAE = 1.08 (C = 1.01 and P = 1.12) over the coupling of T1-w⊕Sagittal. The trained models result in unmatching and distinct learning patterns for healthy and CP patients. Notably, the age-wise brain development-based results deduce a significance in age association at an early age (around two years) and poor at a later age. The study findings will assist neurodevelopmental processing and clinical practices in radiomics without concerning infants’ uncooperative movement or MRI artifacts.

Multi-frame temporal dense nested attention method for detecting GEO objects

The geostationary orbit is a special and important spatial orbital resource. However, space resident objects in this orbit pose potential threats to space security, characterized as dim and small targets in imaging features. To solve the problem of detecting dim and small targets in geostationary orbit, we propose a multi-frame temporal dense nested attention method, which consists of a uniform linear motion constraint module and a dense nested attention module. The dense nested attention module adopts a multi-layer U-shaped structure and channel spatial attention mechanism, which can preserve the deep features of small targets, extract and fuse features from different scales of feature maps, and finally obtain the preliminary detection results of single-frame images. The uniform linear motion constraint module uses the prior information of the target motion in the image sequence, removes false positives and supplements false negatives, and improves the detection performance. Experiments were conducted on the SpotGEO dataset publicly released by the European Space Agency. The results of our experiments indicate that our method outperforms the state-of-the-art approach, showing a 12.83% reduction in Mean Squared Error compared to the current leading method. Our method, characterized by its minimal parameter requirement and independence from extensive dataset pre-training, is ideally suited for integration into small satellite devices, enabling efficient monitoring and mitigation of resident space objects.

A Multi-Level Adaptive Lightweight Net for Damaged Road Marking Detection Based on Knowledge Distillation

To tackle the complexity and limited applicability of high-precision segmentation models for damaged road markings, this study proposes a Multi-level Adaptive Lightweight Network (MALNet) based on knowledge distillation. By incorporating multi-scale dilated convolution and adaptive spatial channel attention fusion modules, the MALNet model significantly enhances the precision, integrity, and robustness of its segmentation branch. Furthermore, it employs an intricate knowledge distillation strategy, channeling rich, layered insights from a teacher model to a student model, thus elevating the latter’s segmentation ability. Concurrently, it streamlines the student model by markedly reducing its parameter count and computational demands, culminating in a segmentation network that is both high-performing and pragmatic. Rigorous testing on three distinct data sets for damaged road marking detection—CDM_P (Collective Damaged road Marking—Public), CDM_H (Collective Damaged road Marking—Highways), and CDM_C (Collective Damaged road Marking—Cityroad)—underscores the MALNet model’s superior segmentation abilities across all damage types, outperforming competing models in accuracy and completeness. Notably, the MALNet model excels in parameter efficiency, computational economy, and throughput. After distillation, the student model’s parameters and computational load decrease to only 31.78% and 27.40% of the teacher model’s, respectively, while processing speeds increase to 1.9 times, demonstrating a significant improvement in lightweight design.

Detecting Artificial Intelligence-Generated images via deep trace representations and interactive feature fusion

The detection of Artificial Intelligence-Generated (AIG) images plays an important role in verifying the authenticity and originality of digital images. However, recent advancements in state-of-the-art image generation methods have significantly challenged the ability to differentiate AIG images from natural photographs (NP). To address this issue, a novel approach based on deep trace representations and dual-branch interactive feature fusion is presented. Firstly, a global feature extraction module that leverages attention-based MobileViT (AT-MobileViT) is designed to learn the deep representations of the global trace information. Besides, we apply multiple enhanced residual blocks to extract discriminative multi-scale features. After that, a low-level feature extraction module incorporating a channel-spatial attention (CSA) block is also carefully employed to enhance the learning of trace representations. To facilitate the capture of complementary information between features, a dual-branch interactive feature fusion module is introduced by reshaping feature vectors into interactive matrices. By conducting experiments on both seen and unseen images, results demonstrate the better performance and robustness of the proposed method.

Attention-Based Deep Learning Approach for Breast Cancer Histopathological Image Multi-Classification.

Breast cancer diagnosis from histopathology images is often time consuming and prone to human error, impacting treatment and prognosis. Deep learning diagnostic methods offer the potential for improved accuracy and efficiency in breast cancer detection and classification. However, they struggle with limited data and subtle variations within and between cancer types. Attention mechanisms provide feature refinement capabilities that have shown promise in overcoming such challenges. To this end, this paper proposes the Efficient Channel Spatial Attention Network (ECSAnet), an architecture built on EfficientNetV2 and augmented with a convolutional block attention module (CBAM) and additional fully connected layers. ECSAnet was fine-tuned using the BreakHis dataset, employing Reinhard stain normalization and image augmentation techniques to minimize overfitting and enhance generalizability. In testing, ECSAnet outperformed AlexNet, DenseNet121, EfficientNetV2-S, InceptionNetV3, ResNet50, and VGG16 in most settings, achieving accuracies of 94.2% at 40×, 92.96% at 100×, 88.41% at 200×, and 89.42% at 400× magnifications. The results highlight the effectiveness of CBAM in improving classification accuracy and the importance of stain normalization for generalizability.

LERFNet: an enlarged effective receptive field backbone network for enhancing visual drone detection

AbstractRecently, the world has witnessed a great increase in drone applications and missions. Drones must be detected quickly, effectively, and precisely when they are being handled illegally. Vision-based anti-drone systems provide an efficient performance compared to radar- and acoustic-based systems. The effectiveness of drone detection is affected by a number of issues, including the drone’s small size, conflicts with other objects, and noisy backgrounds. This paper employs enlarging the effective receptive field (ERF) of feature maps generated from the YOLOv6 backbone. First, RepLKNet is used as the backbone of YOLOv6, which deploys large kernels with depth-wise convolution. Then, to get beyond RepLKNet’s large inference time, a novel LERFNet is implemented. LERFNet uses dilated convolution in addition to large kernels to enlarge the ERF and overcome each other’s problems. The linear spatial-channel attention module (LAM) is used to give more attention to the most informative pixels and high feature channels. LERFNet produces output feature maps with a large ERF and high shape bias to enhance the detection of various drone sizes in complex scenes. The RepLKNet and LERFNet backbones for Tiny-YOLOv6, Tiny-YOLOv6, YOLOv5s, and Tiny-YOLOv7 are compared. In comparison to the aforementioned techniques, the suggested model’s results show a greater balance between accuracy and speed. LERFNet increases the MAP by $$2.8\%$$ 2.8 % , while significantly reducing the GFLOPs and parameter numbers when compared to the original backbone of YOLOv6.

Forest Canopy Height Retrieval Model Based on a Dual Attention Mechanism Deep Network

Accurate estimation of forest canopy height is crucial for biomass inversion, carbon storage assessment, and forestry management. However, deep learning methods are underutilized compared to machine learning. This paper introduces the convolutional neural network–bidirectional long short-term memory (CNN-BiLSTM) model and proposes a Convolutional Neural network–spatial channel attention–bidirectional long short-term memory (CNN-SCA-BiLSTM) model, incorporating dual attention mechanisms for richer feature extraction. A dataset comprising vegetation indices and canopy height data from forest regions in Luoyang, specifically within the 8–20 m range, is used for a comparative analysis of multiple models, with accuracy evaluated based on the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R2). The results demonstrate that (1) the CNN-BiLSTM model exhibits strong potential (MAE = 1.6554 m, RMSE = 2.2393 m, R2 = 0.9115) and (2) the CNN-SCA-BiLSTM model, while slightly less efficient (<1%), demonstrates improved performance. It reduces the MAE by 0.3047 m, the RMSE by 0.6420 m, and increases the R2 value by 0.0495. Furthermore, the model is utilized to generate a canopy height map (MAE = 5.2332 m, RMSE = 7.0426 m) for Henan in the Yellow River Basin for the year 2022. The canopy height is primarily distributed around 5–20 m, approaching the accuracy levels of global maps (MAE = 4.0 m, RMSE = 6.0 m).

Prediction of Operational Noise Uncertainty in Automotive Micro-Motors Based on Multi-Branch Channel–Spatial Adaptive Weighting Strategy

The acoustic performance of automotive micro-motors directly impacts the comfort and driving experience of both drivers and passengers. However, various motor production and testing uncertainties can lead to noise fluctuations during operation. Thus, predicting the operational noise range of motors on the production line in advance becomes crucial for timely adjustments to production parameters and process optimization. This paper introduces a prediction model based on a Multi-Branch Channel–Spatial Adaptive Weighting Strategy (MCSAWS). The model includes a multi-branch feature extraction (MFE) network and a channel–spatial attention module (CSAM). It uses the vibration and noise data from micro-motors’ idle operations on the production line as input to efficiently predict the operational noise uncertainty interval of automotive micro-motors. The model employs the VAE-GAN approach for data augmentation (DA) and uses Gammatone filters to emphasize the noise at the commutation frequency of the motor. The model was compared with Convolutional Neural Networks (CNNs) and Multilayer Perceptrons (MLPs). Experimental results demonstrate that the MCSAWS method is superior to conventional methods in prediction accuracy and reliability, confirming the feasibility of the proposed approach. This research can help control noise uncertainty in micro-motors’ production and manufacturing processes in advance.

An efficient motion visual learning method for video action recognition

Currently, efficient spatio-temporal information modeling is one of the key research components to solve the action recognition problem. Previous approaches focus on enhancing the backbone features individually using hierarchical structures, and unfortunately, most of them fail to achieve a better balance between the interactional adequacy of features within the structure. In this work, we propose an effective Multi-dimensional Adaptive Fusion Network (MDAF-Net), which can be embedded into the mainstream action recognition backbone in a plug-and-play manner to fully activate the transfer and representation of action features in the deep network. Specifically, our MDAF-Net contains two main components: the Adaptive Temporal Capture Module (ATCM) and the Extended Spatial and Channel Module (ESCM). The ATCM effectively suppresses the over-expression of similar features in adjacent frames and activates the expression of motion flow information. The ESCM further improves temporal modeling efficiency by extending the spatial feature perceptual field and enhancing channel attention. Extensive experiments on several challenging action recognition benchmarks, such as Something-Something V1&V2 and Kinetics-400, demonstrate that the proposed MDAF can achieve state-of-the-art and competitive performance.