Lightweight Attention Research Articles

A lightweight object detection framework for underwater imagery with joint image restoration and color transformation

Traditional underwater detectors struggle to match the demands of visual observation and icon detection because of the underwater optical pictures' color distortion, uneven illumination, and excessive noise levels. In this paper, a lightweight deep-water object detection network is proposed, which consists of three key components, namely image restoration, color transformation, and object detection. First, we propose a lightweight attention module for the problems of color distortion and uneven illumination in underwater images. This module pays more attention to the areas with a greater influence of underwater color when dealing with different types of poor-quality images. Second, to reduce the computational complexity of object detection, we propose a color conversion module based on the laplacian pyramid decomposition. This module incorporates multi-scale features to effectively address the loss of details in the brightest or darkest regions. Finally, we propose a multi-scale dense residual object detection network. The network applies a multi-scale kernel to an improved residual module to realize the extraction and fusion of feature information of multiple receptive fields. Furthermore, to deal with the problem of underwater small-sample object detection, we propose a mutually exclusive loss function. The mutual exclusion loss promotes the update of network parameters so that the classes with different characteristics of samples generated by the network are mutually exclusive. We ran tests on the Pycharm platform to show that the suggested lightweight joint underwater object identification algorithm outperformed the most sophisticated ones. The proposed system was put to the test in an intricate underwater setting, achieving 92.56 percent accuracy and 19.2 frames per second.

Global semantic enhancement network for video captioning

Video captioning aims to briefly describe the content of a video in accurate and fluent natural language, which is a hot research topic in multimedia processing. As a bridge between video and natural language, video captioning is a challenging task that requires a deep understanding of video content and effective utilization of diverse video multimodal information. Existing video captioning methods usually ignore the relative importance between different frames when aggregating frame-level video features and neglect the global semantic correlations between videos and texts in learning visual representations, resulting in the learned representations less effective. To address these problems, we propose a novel framework, namely Global Semantic Enhancement Network (GSEN) to generate high-quality captions for videos. Specifically, a feature aggregation module based on a lightweight attention mechanism is designed to aggregate frame-level video features, which highlights features of informative frames in video representations. In addition, a global semantic enhancement module is proposed to enhance semantic correlations for video and language representations in order to generate semantically more accurate captions. Extensive qualitative and quantitative experiments on two public benchmark datasets MSVD and MSR-VTT demonstrate that the proposed GSEN can achieve superior performance than state-of-the-art methods.

A light-weight neural network for marine acoustic signal recognition suitable for fiber-optic hydrophones

Fiber-optic hydrophones based on optical fiber sensing and communication technology have developed rapidly in recent years. With the advantages of high sensitivity, small size, and large dynamic range, they are easy to detect underwater acoustic signals, showing great potential in the field of marine ecological protection. Researchers can monitor the marine ecological environment by recognizing underwater optical fiber sensing signals collected by fiber-optic hydrophones in real-time. Deep learning models can achieve high recognition accuracy, but they usually have large amounts of parameters that makes them difficult to be deployed in mobile terminals of marine survey ships. To solve the key problem, this paper discussed the design of the Deep Light-Weight Attention Residual Network (DLA-ResNet). It introduces the depth-wise separable convolution (DSC) and the convolutional block attention module (CBAM) into ResNet18 to make its performance better. ResNet18 is a two-dimensional convolutional network (CNN) model applying residual blocks. Experiments verify that the accuracy of DLA-ResNet improved from ResNet18 is increased by about 2.04% and its size is reduced by 85.67%. The proposed scheme in this paper realizes an average recognition rate of 96.50% for eight types of underwater optical fiber sensing signals in the application of marine ecological environment monitoring. Moreover, its model size is only 6.12 MB and it only takes 27.90 ms, which is available to meet the needs of reliability and effectiveness.

A lightweight convolutional neural network for recognition and classification for Si3N4 chip substrate surface defects

For the low accuracy and poor efficiency for surface defects detection, identifying and locating defects becomes a key process. However, traditional inspection methods cannot solve these problems. A lightweight improved YOLOX network with attention mechanism (GhostC-YOLOX) for Si3N4 ceramic chip substrate surface defect recognition and classification is proposed. For liner regression, detection and classification tasks, SimOTA label assignment strategy is combined with Moscia and MixUp data augmentation. Dynamic allocation of positive samples is achieved and the dataset is expanded. To the training process, the loss function is optimized. L2 regularization term is introduced to improve the generalization ability. Aiming to generate more feature maps with fewer parameters, the GhostNet structure is embed into the Neck of YOLOX, which reduces the computational cost while maintaining performance. To improve the defect detection accuracy, the lightweight Attention mechanism CBAM(Convolutional Block Attention Module) is integrated into the Backbone tail of YOLOX. The attention is allocated to the two dimensions simultaneously. The detection accuracy of Si3N4 ceramic chip substrate is 96.47%. Compared with YOLOX and Faster RCNN models, the detection accuracy is improved by 2.648% and 18.08% respectively. Moreover, the parameter size of the method is only 5.034 MB. Its detection performance is obviously ahead of other detection methods.

A Survey on Data-Driven Runoff Forecasting Models Based on Neural Networks

As an important branch of time series forecasting, runoff forecasting provides a reliable decision-making basis for the rational use of water resources, economic development and ecological management of river basins. With the revolution of computing power, the data-driven model has become the mainstream runoff forecasting method. This survey will introduce and explore several types of existing neural network for runoff forecasting: convolutional neural network (CNN), recurrent neural network (RNN) and Transformer. The advantages and limitations of these referenced models are also discussed. In addition, this paper also discusses the future improvement directions of runoff forecasting models from the three directions of accuracy, robustness and interpretability. Through plug-and-play lightweight attention mechanism modules, reliable ensemble methods, and forward-looking interpretability methods, the potential of runoff forecasting models can be further tapped to improve the overall performance.

A Novel Residual Gated Recurrent Unit Framework for Runoff Forecasting

Runoff forecasting is the key to the rational use and protection of water resources by mankind. The large-scale application of machine learning and neural networks in hydrological models has made accurate and reliable short-term runoff forecasting possible. In this paper, a novel short-term runoff forecasting framework called ResGRU Plus is proposed with gated recurrent unit (GRU) as the backbone. GRU has the characteristics of long short-term memory (LSTM) that can selectively memorize and forget information while merging gating units to reduce the amount of parameters. Residual network (ResNet) is also deeply integrated with GRU, and its unique shortcut connection effectively solves the degradation problem of traditional neural networks, making it possible to train deep neural networks based on recurrent architecture. Moreover, a lightweight attention mechanism module: squeeze-and-excitation network (SENet) is embedded in the framework. SENet explicitly models the interdependence between feature dimensions through one global average pooling layer (GAP) and two fully-connected layers, and rescales the original features through the learned weights to adaptively amplify or suppress features. Snapshot ensemble method is also used to train ResGRU Plus, which can integrate multiple homogeneous weak learners through one training process to improve the performance of the model at a small cost. In this paper, the hourly runoff of the Columbia River is used as the dataset. The Nash-Sutcliffe coefficient (NSE) of Efficiency and coefficient of determination (R2), which are two common evaluation metrics for hydrological models, are used to measure the performance of the models. Multiple sets of ablation experiments show that the proposed ResGRU Plus, which combines ResNet and attention mechanism, is able to improve depth by a factor of over 4 and accuracy by nearly 18% compared to the vanilla GRU, which further fully validates the effectiveness of combining the residual structure and attention mechanism with the recurrent architecture-based neural network and the feasibility of applying it to runoff forecasting. In addition, several sets of comparative experiments have also demonstrated the state-of-the-art performance of ResGRU Plus with significant improvement in accuracy compared to mainstream time series forecasting models.

Attention-Based Deep Learning Model for Image Desaturation of SDO/AIA

The Atmospheric Imaging Assembly (AIA) onboard the Solar Dynamics Observatory (SDO) captures full-disk solar images in seven extreme ultraviolet wave bands. As a violent solar flare occurs, incoming photoflux may exceed the threshold of an optical imaging system, resulting in regional saturation/overexposure of images. Fortunately, the lost signal can be partially retrieved from non-local unsaturated regions of an image according to scattering and diffraction principle, which is well consistent with the attention mechanism in deep learning. Thus, an attention augmented convolutional neural network (AANet) is proposed to perform image desaturation of SDO/AIA in this paper. It is built on a U-Net backbone network with partial convolution and adversarial learning. In addition, a lightweight attention model, namely criss-cross attention, is embedded between each two convolution layers to enhance the backbone network. Experimental results validate the superiority of the proposed AANet beyond state-of-the-arts from both quantitative and qualitative comparisons.

A Skin Cancer Detector Based on Transfer Learning and Feature Fusion

Background: With the rapid development of advanced artificial intelligence technologies which have been applied in varying types of applications, especially in the medical field. Cancer is one of the biggest problems in medical sciences. If cancer can be detected and treated early, the possibility of a cure will be greatly increased. Malignant skin cancer is one of the cancers with the highest mortality rate, which cannot be diagnosed in time only through doctors’ experience. We can employ artificial intelligence algorithms to detect skin cancer at an early stage, for example, patients are determined whether suffering from skin cancer by detecting skin damage or spots. Objective: We use the real HAM10000 image dataset to analyze and predict skin cancer. Methods: (1) We introduce a lightweight attention module to discover the relationships between features, and we fine-tune the pre-trained model (i.e., ResNet-50) on the HAM10000 dataset to extract the hidden high-level features from the images; (2) we integrate these high-level features with generic statistical features, and use the SMOTE oversampling technique to augment samples from the minority classes; and (3) we input the augmented samples into the XGBoost model for training and predicting. Results: The experimental results show that the accuracy, sensitivity, and specificity of the proposed SkinDet (Skin cancer detector based on transfer learning and feature fusion) model reached 98.24%, 97.84%, and 98.13%. The proposed model has stronger classification capability for the minority classes, such as dermato fibroma and actinic keratoses. Conclusion: SkinDet contains a lightweight attention module and can extract the hidden high-level features of the images by fine-tuning the pretrained model on the skin cancer dataset. In particular, SkinDet integrates high-level features with statistical features and augments samples of these minority classes. Importantly, SkinDet can be applied to classify the samples into minority classes.

A Lightweight Conditional Convolutional Neural Network for Hyperspectral Image Classification

Deep learning (dl), especially convolutional neural networks (cnns), has been proven to be an excellent feature extractor and widely applied to hyperspectral image (hsi) classification. However, dl is a computationally demanding algorithm with many parameters and a high computational burden, which seriously restricts the deployment of dl-based hsi classification algorithms on mobile and embedded systems. In this paper, we propose an extremely lightweight conditional three-dimensional (3D) hsi with a double-branch structure to solve these problems. Specifically, we introduce a lightweight conditional 3D convolution to replace the conventional 3D convolution to reduce the computational and memory cost of the network and achieve flexible hsi feature extraction. Then, based on lightweight conditional 3D convolution, we build two parallel paths to independently exploit and optimize the diverse spatial and spectral features. Furthermore, to precisely locate the key information, which is conducive to classification, a lightweight attention mechanism is carefully designed to refine extracted spatial and spectral features, and improve the classification accuracy with less computation and memory costs. Experiments on three public hsi data sets show that the proposed model can effectively reduce the cost of computation and memory, achieve high execution speed, and better classification performance compared with several recent dl-based models.

Doppler Radar-Based Human Speech Recognition Using Mobile Vision Transformer

As one of the important vital features of the human body, the acquisition of a speech signal plays an important role in human–computer interaction. In this study, voice sounds are gathered and identified using Doppler radar. The skin on the neck vibrates when a person speaks, which causes the vocal cords to vibrate as well. The vibration signal received by the radar will produce a unique micro-Doppler signal according to words with different pronunciations. Following the conversion of these signals into micro-Doppler feature maps, these speech signal maps are categorized and identified. The speech recognition method used in this paper is on neural networks. CNN convolutional neural networks have a lower generalization and accuracy when there are insufficient training samples and sample extraction bias, and the training model is not suitable for use on mobile terminals. MobileViT is a lightweight transformers-based model that can be used for image classification tasks. MobileViT uses a lightweight attention mechanism to extract features with a faster inference speed and smaller model size while ensuring a higher accuracy. Our proposed method does not require large-scale data collection, which is beneficial for different users. In addition, the learning speed is relatively fast, with an accuracy of 99.5%.

MineSDS: A Unified Framework for Small Object Detection and Drivable Area Segmentation for Open-Pit Mining Scenario.

To tackle the challenges posed by dense small objects and fuzzy boundaries on unstructured roads in the mining scenario, we proposed an end-to-end small object detection and drivable area segmentation framework for open-pit mining. We employed a convolutional network backbone as a feature extractor for both two tasks, as multi-task learning yielded promising results in autonomous driving perception. To address small object detection, we introduced a lightweight attention module that allowed our network to focus more on the spatial and channel dimensions of small objects without impeding inference time. We also used a convolutional block attention module in the drivable area segmentation subnetwork, which assigned more weight to road boundaries to improve feature mapping capabilities. Furthermore, to improve our network perception accuracy of both tasks, we used weighted summation when designing the loss function. We validated the effectiveness of our approach by testing it on pre-collected mining data which were called Minescape. Our detection results on the Minescape dataset showed 87.8% mAP index, which was 9.3% higher than state-of-the-art algorithms. Our segmentation results surpassed the comparison algorithm by 1 percent in MIoU index. Our experimental results demonstrated that our approach achieves competitive performance.

NAMRTNet: Automatic Classification of Sleep Stages Based on Improved ResNet-TCN Network and Attention Mechanism

Sleep, as the basis for regular body functioning, can affect human health. Poor sleep conditions can lead to various physical ailments, such as poor immunity, memory loss, slow cognitive development, and cardiovascular diseases. Along the increasing stress in society comes with a growing surge in conditions associated with sleep disorders. Studies have shown that sleep stages are essential for the body’s memory, immune system, and brain functioning. Therefore, automatic sleep stage classification is of great medical practice importance as a basis for monitoring sleep conditions. Although previous research into the classification of sleep stages has been promising, several challenges remain to be addressed: (1) The EEG signal is a non-smooth signal with harrowing feature extraction and high requirements for model accuracy. (2) Some existing network models suffer from overfitting and gradient descent. (3) Correlation between long time sequences is challenging to capture. This paper proposes NAMRTNet, a deep model architecture based on the original single-channel EEG signal to address these challenges. The model uses a modified ResNet network to extract features from sub-epochs of individual epochs, a lightweight attention mechanism normalization-based attention module (NAM) to suppress insignificant features, and a temporal convolutional network (TCN) network to capture dependencies between features of long time series. The recognition rate of 20-fold cross-validation with the NAMRTNet model for Fpz-cz channel data in the public sleep dataset Sleep-EDF was 86.2%. The experimental results demonstrate the network’s superiority in this paper, surpassing some state-of-the-art techniques in different evaluation metrics. Furthermore, the total time to train the network was 5.1 h, which was much less than the training time of other models.

AGHRNet: An attention ghost-HRNet for confirmation of catch‐and‐shake locations in jujube fruits vibration harvesting

The development of an intelligent jujube fruit harvesting device is a critical step in achieving the whole mechanization process. Catch‐and‐shake harvesting, as an efficient and stable vibration harvesting method, has widely been used to save labor and improve harvesting efficiency in large-scale jujube orchards. However, existing catch‐and‐shake harvesters still rely heavily on the operator's naked eyes to determine the shaking position, which is subjective, highly inefficient and highly labor intensive. To address this problem, this study proposes a computer vision system including truck and branches segmentation, skeleton extraction and vibration position selection to automatically identify appropriate shaking positions. A lightweight attention Ghost-HRNet (AGHRNet) based on deep learning is designed to separate the truck and branches from the complex orchard background. In AGHRNet, an attention Ghost block is proposed to lighten the model and improve segmentation accuracy. Moreover, we also construct a hybrid loss function to solve the pixel-level category imbalance of the trunks, branches and background. To verify the effectiveness of the AGHRNet, ten state-of-the-art semantic segmentation methods, including the FCN8s, SegNet, U-Net, PSPNet, DeepLabv3+, BiSeNet, DDRNet, DANet, SegFormer and HRNet, are compared. Comparative experimental results show that compared with the other state-of-the-art methods, the proposed AGHRNet has higher segmentation accuracy (77.79 % mIoU, 89.46 % mPA) and a smaller model size (9.53 MB) on our datasets. Finally, mask medial axis extraction and the minimum inscribed circle operation are applied to obtain the appropriate vibration location. The computer vision system can provide technical references for catch‐and‐shake harvesting and developing jujube fruit-harvesting robots.

An efficient lightweight neural network using BiLSTM-SCN-CBAM with PCA-ICEEMDAN for diagnosing rolling bearing faults

This study proposes an efficient rolling bearing fault diagnosis model of a hybrid neural network with a lightweight attention mechanism. Firstly, to achieve the low complexity of deep learning (DL) computation, data reduction and denoising are performed by sparse convolutional network (principal component analysis and improved complete ensemble empirical modal decomposition of adaptive noise), then processed data is imported to the hybrid neural network model with convolutional block attention module. The bi-directional long short-term memory and sparse convolutional networks are used in the backbone of the model. A lightweight, generalized attention mechanism is introduced to the last layer of the model for enhancing feature learning, which can further improve the diagnostic accuracy and efficiency. Compared with existing DL fault diagnosis models, In simulating the most realistic cross-conditions and cross-platform conditions, which leads to the random nature of fault generation and makes model diagnosis more complex, the proposed method still maintains less running time and excellent diagnostic accuracy. Finally, the experimental results fully prove that the model has reliable robust and efficient, and it achieves the best balance of diagnostic accuracy and diagnostic efficiency of the hybrid DL model.

DOLG-NeXt: Convolutional neural network with deep orthogonal fusion of local and global features for biomedical image segmentation

Biomedical image segmentation (BMIS) is an essential yet challenging task for the visual analysis of biomedical images. Modern deep learning-based architectures, such as UNet, UNet-based variants, Transformers-based networks, and their combinations, have achieved reasonable success in BMIS. However, they still face certain shortcomings in extracting fine-grained features. They are also limited by scenarios where the modeling of local and global feature representations needs to be optimized correctly for spatial dependency in the decoding process, which can result in duplicate data utilization throughout the architecture. Besides, Transformer-based models lack inductive bias in addition to the complexity of the models. As a result, it can perform unsatisfactorily in a lesser biomedical image setting. This paper proposes a novel encode-decoder architecture named DOLG-NeXt, incorporating three major enhancements over the UNet-based variants. Firstly, we integrate squeeze and excitation network (SE-Net)-driven ConvNeXt stages as encoder backbone for effective feature extraction. Secondly, we employ a deep orthogonal fusion of local and global (DOLG) features module in the decoder to retrieve fine-grained contextual feature representations. Finally, we construct a SE-Net-like lightweight attention network alongside the DOLG module to provide refined target-relevant channel-based feature maps for decoding. To objectively validate the proposed DOLG-NeXt method, we perform extensive quantitative and qualitative analysis on four benchmark datasets from different biomedical image modalities: colonoscopy, electron microscopy, fluorescence, and retinal fundus imaging. DOLG-NeXt achieves a dice coefficient score of 95.10% in CVC-ClinicDB, 95.80% in ISBI 2012, 94.77% in 2018 Data Science Bowl, and 84.88% in the DRIVE dataset, respectively. The experimental analysis shows that DOLG-NeXt outperforms several state-of-the-art models for BMIS tasks.

Information complementary attention-based multidimension feature learning for person re-identification

With the need for criminal investigation technology and the development of deep learning, the task of person re-identification has gradually become a research hotspot. Recently, various neural network-based person re-identification technologies designed by researchers have shown excellent results. However, most of the frameworks focus on complex structural design or redundant networks to guide model construction, which hugely increases the cost of train and application cost. In addition, the correlation between the channel information and spatial information on the pedestrian feature map is also relatively lacking. Therefore, we design a lightweight attention module to address the lack of correlation question response. The proposed module sequentially extracts person images’ channel and spatial features and effectively associates the two kinds of information through sequential connections. The proposed attention module has a simple structure, and the parameter increase in the backbone network is tiny. We place the fuse module in each feature extraction layer to focus on the pedestrian information extracted by each layer. To solve the problem of complex model structure, we choose the residual network as the backbone network and the attention mechanism to extract person features without using pose point estimation or additional network assistance to reduce model complexity. We adjust the drop rate of the person classification layer to improve the model’s generalization ability. We estimate the performance of our method on three public datasets: Market-1501, DukeMTMC-reID, and CUHK03 (both detected and labeled) demonstrate the proposed method’s effectiveness and obtain highly competitive performance on the three datasets.

Lightweight feature point detection network with channel enhancement

Learning-based local feature point extraction work has made great research progress in related computer vision tasks. However, it is difficult to balance the detection speed and detection efficiency of general deep neural networks, which are affected by illumination and viewpoint changes in outdoor environments. To this end, a lightweight image feature point efficient detection network jointly trained by detector–descriptor is proposed. The shared encoding of the feature map is carried out through the multi-branch structure, and a shuffle block model is designed to enhance the channel features. The enhanced feature is divided into three branches to further encode the feature map. One branch improves the network’s ability to identify feature information through a lightweight attention mechanism, and the other two branches perform feature aggregation on the detector and the descriptor through skip connection layers to enrich image features. Considering the problem of network operation efficiency, the multi-branch coding structure is coupled into a set of parameters corresponding to the plane forward propagation structure through the structure re-parameterization technology, which can reduce the complexity of the network and improve the inference speed of the network. In addition, the correct convergence of the network is guaranteed by constraints on the classification loss and position loss, and the sparsely sampled descriptors are used for training to reduce the computational cost. Experimental results on the Hpatches and KITTI datasets show that the proposed method reduces the parameters and GFLOPs by 10.7% and 61.6%, respectively, compared with recent lightweight work, while maintaining comparable accuracy. Moreover, the proposed method achieves a detection speed of 55FPS on images with a resolution of 480×640. https://github.com/SpiritAshes/Channel-Enhancement.

A sparse lightweight attention network for image super-resolution

A sparse lightweight attention network for image super-resolution

EBUNet: a fast and accurate semantic segmentation network with lightweight efficient bottleneck unit

It has been difficult to achieve a suitable balance between effectiveness and efficiency in lightweight semantic segmentation networks in recent years. The goal of this work is to present an efficient and reliable semantic segmentation method called EBUNet, which is aimed at achieving a favorable trade-off between inference speed and prediction accuracy. Initially, we develop an Efficient Bottleneck Unit (EBU) that employs depth-wise convolution and depth-wise dilated convolution to obtain adequate features with moderate computation costs. Then, we developed a novel Image Partition Attention Module (IPAM), which divides feature maps into subregions and generates attention weights based on them. As a third step, we developed a novel lightweight attention decoder with which to retrieve spatial information effectively. Extensive experiments show that our EBUNet achieves 73.4% mIou and 152 FPS on the Cityscapes dataset and 72.2% mIoU and 147 FPS on the Camvid dataset with only 1.57 M parameters. The results of the experiment confirm that the proposed model is capable of making a decent trade-off in terms of accuracy, inference, and model size. The source code of our EBUNet is available at (https://github.com/Skybird1101/EBUNet).

PolypSeg+: A Lightweight Context-Aware Network for Real-Time Polyp Segmentation.

Automatic polyp segmentation from colonoscopy videos is a prerequisite for the development of a computer-assisted colon cancer examination and diagnosis system. However, it remains a very challenging task owing to the large variation of polyps, the low contrast between polyps and background, and the blurring boundaries of polyps. More importantly, real-time performance is a necessity of this task, as it is anticipated that the segmented results can be immediately presented to the doctor during the colonoscopy intervention for his/her prompt decision and action. It is difficult to develop a model with powerful representation capability, yielding satisfactory segmentation results and, simultaneously, maintaining real-time performance. In this article, we present a novel lightweight context-aware network, namely, PolypSeg+, attempting to capture distinguishable features of polyps without increasing network complexity and sacrificing time performance. To achieve this, a set of novel lightweight techniques is developed and integrated into the proposed PolypSeg+, including an adaptive scale context (ASC) module equipped with a lightweight attention mechanism to tackle the large-scale variation of polyps, an efficient global context (EGC) module to promote the fusion of low-level and high-level features by excluding background noise and preserving boundary details, and a lightweight feature pyramid fusion (FPF) module to further refine the features extracted from the ASC and EGC. We extensively evaluate the proposed PolypSeg+ on two famous public available datasets for the polyp segmentation task: 1) Kvasir-SEG and 2) CVC-Endoscenestill. The experimental results demonstrate that our PolypSeg+ consistently outperforms other state-of-the-art networks by achieving better segmentation accuracy in much less running time. The code is available at https://github.com/szu-zzb/polypsegplus.