Multi-level Attention Research Articles

A light-weight enhanced multi-level attention network for plant disease identification

A light-weight enhanced multi-level attention network for plant disease identification

Multilevel attention mechanism for motion fatigue recognition based on sEMG and ACC signal fusion.

This study aims to develop a cost-effective and reliable motion monitoring device capable of comprehensive fatigue analysis. It achieves this objective by integrating surface electromyography (sEMG) and accelerometer (ACC) signals through a feature fusion strategy. The study introduces a multi-level attention mechanism for classification, leveraging convolutional neural networks (CNNs). The preprocessing phase involves a local feature attention mechanism that enhances local waveform features using the amplitude envelope. A dual-scale attention mechanism, operating at both channel and neuron levels, is employed to enhance the model's learning from high-dimensional fused data, improving feature extraction and generalization. The local feature attention mechanism significantly improves the model's classification accuracy and convergence, as demonstrated in ablation experiments. The model, optimized with multi-level attention mechanisms, excels in accuracy and generalization, particularly in handling data with pseudo-artifacts. Computational analysis indicates that the proposed optimization algorithm has minimal impact on CNN's training and testing times. The study achieves recognition accuracies of 92.52%, 92.38%, and 92.30%, as well as F1-scores of 91.92%, 92.13%, and 92.29% for the three fatigue states, affirming its reliability. This research provides technical support for the development of affordable and dependable wearable motion monitoring devices.

Multi-relation spatiotemporal graph residual network model with multi-level feature attention: A novel approach for landslide displacement prediction

Accurate prediction of landslide displacement is crucial for effective early warning of landslide disasters. While most existing prediction methods focus on time-series forecasting for individual monitoring points, there is limited research on the spatiotemporal characteristics of landslide deformation. This paper proposes a novel Multi-Relation Spatiotemporal Graph Residual Network with Multi-Level Feature Attention (MFA-MRSTGRN) that effectively improves the prediction performance of landslide displacement through spatiotemporal fusion. This model integrates internal seepage factors as data feature enhancements with external triggering factors, allowing for accurate capture of the complex spatiotemporal characteristics of landslide displacement and the construction of a multi-source heterogeneous dataset. The MFA-MRSTGRN model incorporates dynamic graph theory and four key modules: multi-level feature attention, temporal-residual decomposition, spatial multi-relational graph convolution, and spatiotemporal fusion prediction. This comprehensive approach enables the efficient analyses of multi-source heterogeneous datasets, facilitating adaptive exploration of the evolving multi-relational, multi-dimensional spatiotemporal complexities in landslides. When applying this model to predict the displacement of the Liangshuijing landslide, we demonstrate that the MFA-MRSTGRN model surpasses traditional models, such as random forest (RF), long short-term memory (LSTM), and spatial temporal graph convolutional networks (ST-GCN) models in terms of various evaluation metrics including mean absolute error (MAE = 1.27 mm), root mean square error (RMSE = 1.49 mm), mean absolute percentage error (MAPE = 0.026), and R-squared (R2 = 0.88). Furthermore, feature ablation experiments indicate that incorporating internal seepage factors improves the predictive performance of landslide displacement models. This research provides an advanced and reliable method for landslide displacement prediction.

MGFNet: Cross-scene Crowd Counting via Multistage Gated Fusion Network

Existing crowd counting methods are mainly trained and tested in similar scenarios. When the testing and training scenarios of the model are different, the counting accuracy of these methods will sharply decrease, which seriously limits their practical application. To address this problem, we propose a multistage gated fusion network (MGFNet) for cross-scene crowd counting. MGFNet is primarily composed of dynamic gated convolution units (DGCU) and multilevel scale attention blocks (MSAB) modules. Specifically, DGCU uses a dynamic gating path to supplement detailed information to reduce the loss of crowd information and overestimation of background in different scenarios. MSAB calibrates crowd information at different scales and perspectives in different scenes by generating attention maps with discriminative information. In addition, we used a new global local consistency loss to optimize the model to adapt to changes in crowd density and distribution. Extensive experiments on four different types of scene counting benchmarks show that the proposed MGFNet achieves superior cross-scene counting performance.

A multilevel cooperative attention network of precise quantitative analysis for predicting ractopamine concentration via adaptive weighted feature selection and multichannel feature fusion

Surface enhanced Raman spectroscopy (SERS) holds great potential due to its rapid detection and high sensitivity. However, issues such as signal noise, fluctuations, and spectral shifts can negatively impact its performance in detecting ractopamine in pork. Hierarchical Gradient Aware Spectral Network (HGASNet) was proposed to address these issues. The key innovations are the Spectral Gradient Weighted Attention (SGWA) and Multi-Channel Peak Attention Mechanism (MCPAM) modules within HGASNet. The SGWA module dynamically adjusts feature weights, enhancing sensitivity to critical Raman spectral shifts. Meanwhile, MCPAM leverages multi-channel attention to better capture long range dependencies and fuse global and local information. Additionally, HGASNet's hierarchical structure incrementally extracts and integrates features at various levels, enabling the model to focus on global features in the higher layers while preserving fine-grained local details in the lower layers. Experimental results show HGASNet outperforming existing approaches, achieving R2 of 0.9972, RMSRE of 0.0413, RMSLE of 0.0548, sMAPE of 6.47 %, and MAPE of 6.72 %.

An effective multi-scale interactive fusion network with hybrid Transformer and CNN for smoke image segmentation

Smoke has visually elusive appearances, especially in low-light conditions, so it is quite difficult to quickly and accurately detect smoke from images. To address these challenges, we design a dual-encoder structure of Transformer and Convolutional Neural Network (CNN) to propose an effective Multi-scale Interactive Fusion Network (MIFNet) for smoke image segmentation. To enhance the presentation of features, we propose a Local Feature Enhancement Propagation (LFEP) module to enhance spatial details. To optimize global and local features for efficient fusion, we integrate LFEP into the original Transformer to replace the traditional multi-head self-attention mechanism. Then, we propose a Multi-level Attention Coupled Module (MACM) to fuse Transformer and CNN features of the dual-encoder. MACM can flexibly focus on information interaction between different levels of two encoding paths. Finally, we design a Prior-guided Multi-scale Fusion Decoder (PMFD), which combines prior knowledge with a multi-scale feature fusion strategy to improve the performance of segmentation. Experimental results demonstrate that MIFNet substantially outperforms the state-of-the-art methods. MIFNet achieves a mean Intersection over Union (mIoU) of 81.6% on the synthetic smoke (SYN70K) dataset, and a remarkable accuracy of 98.3% on the forest smoke dataset.

Multi-level attention graph feature fusion smooth prognostics approach for aircraft engines remaining useful life prediction

In the development of a prognostic and health management system, it is a challenge to mine the degradation characteristics, estimate the RUL reliably and smoothly, and explain the RUL prediction results from the multi-sensor condition monitoring data collected under the non-fixed-length equipment operation cycle and multi-operating conditions. In this paper, a Multi-level Attention Graph feature Fusion Smooth Prognostics Approach (MAGF-SPA) is proposed to achieve accurate, smooth, and interpretable RUL prediction results. In this method, firstly, multivariate time series (MTS) are transformed into graph signals from a purely data-driven perspective. Secondly, the four-level feature extraction strategy, which includes node-level, subgraph-level, cyclical-level, and temporal-level feature extraction, is designed to extract the inherent hidden degradation characteristics of MTS. Finally, a smoothing trick proposed in this paper can optimise the RUL prediction value corresponding to the degradation characteristics in real time. Experimental results show that the proposed method achieves state-of-the-art on the N-CMAPSS dataset.

Advancing Session-Based Recommendations with Atten-Mixer+: Dynamic and Adaptive Multi-Level Intent Mining

Session-based recommendation (SBR) systems, traditionally reliant on complex graph neural networks (GNNs), often face challenges with marginal performance improvements despite increased model complexity. In this paper, we dissect the classical GNN-based SBR models and empirically find that the sophisticated GNN propagations might be redundant, given the readout module plays a significant role in GNN-based models. Based on this observation, we introduce Atten-Mixer+, an advanced iteration of our previously developed Multi-Level Attention Mixture Network (Atten-Mixer). Atten-Mixer+ forgoes GNN propagation in favor of a dynamic and adaptive readout process, tailored to the unique characteristics of each session. Different from the vanilla version, Atten-Mixer+ features the Adaptive Intent Scaler (AIS) layer, which dynamically determines the depth of multi-level user intent analysis, and a soft allocation approach for generating user intent queries across entire user interaction sequences. This innovative design allows Atten-Mixer+ to capture a nuanced and comprehensive understanding of user behaviors, overcoming the limitations of fixed-length analysis. Empirical evaluations on benchmark datasets highlight Atten-Mixer+'s superior efficiency and effectiveness, marking a significant step forward in the predictive accuracy of session-based recommendation systems.

Research on Animation Character Expression Generation Based on Attention Conditioned Cyclegan

With advancements in computer vision and AI, deep learning is applied in animation and game design to enhance facial expression realism. This study proposes ACCGAN, an innovative method for automated animated character expression generation and transfer using deep learning. By integrating cycle consistency, conditional generation, and attention mechanisms, ACCGAN optimizes GANs in capturing complex expression features, maintaining naturalness, and accurately expressing details. ACCGAN utilizes multi-level attention mechanisms, dynamically adjusting focus on key facial features, enhancing capture and expression.

MATNet: Multilevel attention-based transformers for change detection in remote sensing images

Remote sensing image change detection is crucial for natural disaster monitoring and land use change. As the resolution increases, the scenes covered by remote sensing images become more complex, and traditional methods have difficulties in extracting detailed information. With the development of deep learning, the field of change detection has new opportunities. However, existing algorithms mainly focus on the difference analysis between bi-temporal images, while ignoring the semantic information between images, resulting in the global and local information not being able to interact effectively. In this paper, we introduce a new transformer-based multilevel attention network (MATNet), which is capable of extracting multilevel features of global and local information, enabling information interaction and fusion, and thus modeling the global context more effectively. Specifically, we extract multilevel semantic features through the Transformer encoder, and utilize the Feature Enhancement Module (FEM) to perform feature summing and differencing on the multilevel features in order to better extract the local detail information, and thus better detect the changes in small regions. In addition, we employ a multilevel attention decoder (MAD) to obtain information in spatial and spectral dimensions, which can effectively fuse global and local information. In experiments, our method performs excellently on CDD, DSIFN-CD, LEVIR-CD, and SYSU-CD datasets, with F1 scores and OA reaching 95.67%∕87.75%∕90.94%∕86.82% and 98.95%∕95.93%∕99.11%∕90.53% respectively.

Trajectory representation learning with multilevel attention for driver identification

Massive trajectory data have originated from the development of positioning technology. Learning GPS trajectory representation to characterize a driver’s driving style is a challenging task with important applications in many areas, including autonomous driving, auto insurance, advanced driver assistance systems, urban computing, and the internet of things. Few studies have considered the interactions between different factors. In this study, we propose a novel trajectory representation method based on a multilevel attention mechanism (ATTraj2vec) and apply it to the task of driver identification. We use 1D CNN to summarize local features, including motion, spatial and temporal features. Then we utilize a multilevel attention mechanism to extract global features, aggregating the interactions of motion features with temporal and spatial features progressively. Additionally, we adopt multi-loss to optimize our model simultaneously, which consists of a softmax loss for driver classification and Siamese loss for making trajectories from the same driver more similar. Classification experimental results on a real-world automobile trajectory dataset demonstrated that our proposed model significantly outperforms existing baselines. Meanwhile, the proposed method provides significant gains in the trajectory clustering of unseen drivers. The code is available in the repository, https://github.com/limengyuan2021/ATTraj2vec,

Adaptive multilevel attention deeplabv3+ with heuristic based frame work for semantic segmentation of aerial images using improved golden jackal optimization algorithm

Semantic segmentation of small-scale objects in aerial images is challenging due to low-level characteristics in neural networks and varying information in feature maps. Deeplab models struggle with poor edge refinement, resulting in rough borders, and fail to fully exploit relationships between pixel categories at different distances. To addressing the issue in deeplab series, an adaptive multi-level attention based deeplabv3+ (AMLA-Deeplabv3+) with improved golden jackal optimization algorithm is implemented in this paper. Multi-level attention unit has been included in the Atrous spatial pyramid pooling module in the encoder section of deeplabv3+ to bridge the semantic feature gap among encoders output. To put more weights on relevant features squeeze and excitation units has been included in the decoder section of deeplabv3+. The improved golden jackal optimization (IGJO) algorithm is deployed to fine tune the hyper parameters such as number of hidden neurons, epochs, learning rate and batch size of deeplabv3+. The proposed AMLA-Deeplabv3+ is tested on two publicly available datasets named semantic segmentation of aerial imagery and aerial image segmentation. The proposed model achieves better segmentation outcomes in terms of accuracy of 99.65%, Precision of 97.21%, F1-score of 99.48% and mean intersection over union of 98.44% with a computation time of 3.0231 minutes. The experimental outcomes shows that the proposed model’s efficiency is superior to other conventional techniques.

Frequency aware high-quality computer-generated holography via multilevel wavelet learning and channel attention.

Deep learning-based computer-generated holography offers significant advantages for real-time holographic displays. Most existing methods typically utilize convolutional neural networks (CNNs) as the basic framework for encoding phase-only holograms (POHs). However, recent studies have shown that CNNs suffer from spectral bias, resulting in insufficient learning of high-frequency components. Here, we propose a novel, to our knowledge, frequency aware network for generating high-quality POHs. A multilevel wavelet-based channel attention network (MW-CANet) is designed to address spectral bias. By employing multi-scale wavelet transformations, MW-CANet effectively captures both low- and high-frequency features independently, thus facilitating an enhanced representation of high-frequency information crucial for accurate phase inference. Furthermore, MW-CANet utilizes an attention mechanism to discern and allocate additional focus to critical high-frequency components. Simulations and optical experiments confirm the validity and feasibility of our method.

Quaternion attention-based JND model for macrophotography image watermarking

With the advancement of imaging technology, macrophotography images (MPIs) have become a popular research topic. Unlike natural images, MPIs often feature sharp foregrounds and blurred backgrounds, leading to distinct perceptual characteristics in estimation. As the number of MPIs grows rapidly, concerns over image quality and security increase. Robust watermarking techniques have been introduced to address these challenges. Just Noticeable Difference (JND) has been widely used in quantization-based watermarking frameworks. However, existing JND models handle each image area with a single-level perceptual attention. Visual attention in Quaternion Discrete Wavelet Transform (QDWT), which can reflect the Multi-level perceptual attention feature. Therefore, we propose a new method called Quaternion Attention-based Just Noticeable Difference model for MPIs Watemarking (QAJnd-MW) for watermarking MPIs. This method uses visual attention mechanisms, recognizing that the HVS is more sensitive to attention regions. We generate a masking effect in the JND field. The input image undergoes QDWT to explore multi-scale features. The multi-scale feature maps, with multi-directional luminance and multi-channel color, help create local and global attention maps, which are fused to form the final attention map. Specifically, considering both attention-based masking effects, the quaternion attention-guided JND model is designed for a robust MPI watermarking framework, aiming to further improve MPI security. Extensive experiments on the MP2020 and Blur Detection datasets show that the proposed model significantly improves robustness against JPEG compression attacks, reducing the bit error rate (BER) by up to 12%. Additionally, the model performs well against other attacks, such as those in online social networks, with lower BER than current state-of-the-art techniques.

Multilevel attention imitation knowledge distillation for RGB-thermal transmission line detection

Transmission line detection (TLD) plays a crucial role in ensuring the safety and stability of electricity supply. Applying RGB-thermal convolutional neural networks (CNNs) to unmanned aerial vehicles (UAVs) photography is a valuable alternative for diagnosing transmission line faults. However, existing CNNs struggle to generalize to TLD due to the clustered backgrounds and variable weather conditions. In addition, the limited computational resources and storage space of UAVs pose challenges for the lightweight design of models. In the present study, we developed a novel multilevel attention imitation knowledge distillation structure comprising a high-performing teacher model called MAINet-T and a compact student model called MAINet-S. We aimed to 1) improve the accuracy and robustness of TLD and 2) optimize the performance and capacity of the model for deployment on UAVs. The MAINet-T has a three-stage feature aggregation module and a detailed enhancement module to facilitate the processes of multi-modal and multilevel feature complement and interaction. To balance model performance and capacity for deployment, we proposed a novel KD strategy, including response distillation and feature distillation, to obtain an optimized model called MAINet-S*. Within feature distillation, we proposed a multilevel attention imitation module to integrate the advantages of the attention maps in different stages of the encoder. In experiments based on the VITLD dataset, MAINet-S* outperformed 15 state-of-the-art methods, with a 66.2% reduction in the number of weight parameters (Params) and a 69.9% increase in floating-point operations (FLOPs) compared with MAINet-T.

A contrastive topic-aware attentive framework with label encodings for post-disaster resource classification

Social media has emerged as a critical platform for disseminating real-time information during disasters. However, extracting actionable resource data, such as needs and availability, from this vast and unstructured content remains a significant challenge, leading to delays in identifying and allocating resources, with severe consequences for affected populations. This study addresses this challenge by investigating the potential of label and topic features, combined with text embeddings, to enhance the performance and efficiency of resource identification from social media data. We propose Crisis Resource Finder (CRFinder), a novel framework that leverages label encoding and topic features to extract richer contextual information, uncover hidden patterns, and reveal the true context of disaster resources. CRFinder incorporates novel techniques such as multi-level text-label attention and contrastive text-topic attention to capture semantic and thematic nuances within the textual data. Additionally, our approach employs topic injection and selective contextualization techniques to enhance thematic relevance and focus on critical information, which is pivotal for targeted relief efforts. Extensive experiments demonstrate the significant improvements achieved by CRFinder over existing state-of-the-art methods, with average weighted F1-score gains of 7.12%, 6.44%, and 7.89% on datasets from the Nepal earthquake, Italy earthquake, and Chennai floods, respectively. By providing timely and accurate insights into resource needs and availabilities, CRFinder has the potential to revolutionize disaster response efforts.

TDFFM: Transformer and Deep Forest Fusion Model for Predicting Coronavirus 3C-Like Protease Cleavage Sites.

COVID-19, caused by the highly contagious SARS-CoV-2 virus, is distinguished by its positive-sense, single-stranded RNA genome. A thorough understanding of SARS-CoV-2 pathogenesis is crucial for halting its proliferation. Notably, the 3C-like protease of the coronavirus (denoted as 3CLpro) is instrumental in the viral replication process. Precise delineation of 3CLpro cleavage sites is imperative for elucidating the transmission dynamics of SARS-CoV-2. While machine learning tools have been deployed to identify potential 3CLpro cleavage sites, these existing methods often fall short in terms of accuracy. To improve the performances of these predictions, we propose a novel analytical framework, the Transformer and Deep Forest Fusion Model (TDFFM). Within TDFFM, we utilize the AAindex and the BLOSUM62 matrix to encode protein sequences. These encoded features are subsequently input into two distinct components: a Deep Forest, which is an effective decision tree ensemble methodology, and a Transformer equipped with a Multi-Level Attention Model (TMLAM). The integration of the attention mechanism allows our model to more accurately identify positive samples, thus enhancing the overall predictive performance. Evaluation on a test set demonstrates that our TDFFM achieves an accuracy of 0.955, an AUC of 0.980, and an F1-score of 0.367, substantiating the model's superior prediction capabilities.

Context enhanced multi-level attention model for session-based recommendation

Context enhanced multi-level attention model for session-based recommendation

Dynamic traffic network representation model for improving the prediction performance of passenger flow for mass rapid transit

Accurate machine learning predictions of passenger flow data for mass rapid transit (MRT) systems can considerably improve operational efficiency by enabling better allocation of train and human resources. However, such predictions are challenging because MRT networks have complex structures with route dependence and transfer stations. Although the static state of an MRT network has been computed in previous studies, a comprehensive understanding of an MRT network requires characterizing its dynamics. Therefore, this paper proposes a dynamic traffic network representation (DTNR) model that captures station features from historical traffic flows and geographical information of MRT stations. Furthermore, a multilevel attention network (MLAN) model is proposed to predict MRT passenger flow as a downstream task following the pretraining of the DTNR model. The experimental results of this study indicate that the developed DTNR and MLAN models can accurately predict MRT passenger flow. These models are widely applicable to different MRT systems and passenger flow situations, making them a valuable tool for transportation planners and operators.

Revolutionizing health monitoring: Integrating transformer models with multi-head attention for precise human activity recognition using wearable devices.

A daily activity routine is vital for overall health and well-being, supporting physical and mental fitness. Consistent physical activity is linked to a multitude of benefits for the body, mind, and emotions, playing a key role in raising a healthy lifestyle. The use of wearable devices has become essential in the realm of health and fitness, facilitating the monitoring of daily activities. While convolutional neural networks (CNN) have proven effective, challenges remain in quickly adapting to a variety of activities. This study aimed to develop a model for precise recognition of human activities to revolutionize health monitoring by integrating transformer models with multi-head attention for precise human activity recognition using wearable devices. The Human Activity Recognition (HAR) algorithm uses deep learning to classify human activities using spectrogram data. It uses a pretrained convolution neural network (CNN) with a MobileNetV2 model to extract features, a dense residual transformer network (DRTN), and a multi-head multi-level attention architecture (MH-MLA) to capture time-related patterns. The model then blends information from both layers through an adaptive attention mechanism and uses a SoftMax function to provide classification probabilities for various human activities. The integrated approach, combining pretrained CNN with transformer models to create a thorough and effective system for recognizing human activities from spectrogram data, outperformed these methods in various datasets - HARTH, KU-HAR, and HuGaDB produced accuracies of 92.81%, 97.98%, and 95.32%, respectively. This suggests that the integration of diverse methodologies yields good results in capturing nuanced human activities across different activities. The comparison analysis showed that the integrated system consistently performs better for dynamic human activity recognition datasets. In conclusion, maintaining a routine of daily activities is crucial for overall health and well-being. Regular physical activity contributes substantially to a healthy lifestyle, benefiting both the body and the mind. The integration of wearable devices has simplified the monitoring of daily routines. This research introduces an innovative approach to human activity recognition, combining the CNN model with a dense residual transformer network (DRTN) with multi-head multi-level attention (MH-MLA) within the transformer architecture to enhance its capability.