Attention Mechanism Research Articles

Deep Reinforcement Learning with Local Attention for Single Agile Optical Satellite Scheduling Problem

This paper investigates the single agile optical satellite scheduling problem, which has received increasing attention due to the rapid growth in earth observation requirements. Owing to the complicated constraints and considerable solution space of this problem, the conventional exact methods and heuristic methods, which are sensitive to the problem scale, demand high computational expenses. Thus, an efficient approach is demanded to solve this problem, and this paper proposes a deep reinforcement learning algorithm with a local attention mechanism. A mathematical model is first established to describe this problem, which considers a series of complex constraints and takes the profit ratio of completed tasks as the optimization objective. Then, a neural network framework with an encoder–decoder structure is adopted to generate high-quality solutions, and a local attention mechanism is designed to improve the generation of solutions. In addition, an adaptive learning rate strategy is proposed to guide the actor–critic training algorithm to dynamically adjust the learning rate in the training process to enhance the training effectiveness of the proposed network. Finally, extensive experiments verify that the proposed algorithm outperforms the comparison algorithms in terms of solution quality, generalization performance, and computation efficiency.

On-site burn severity assessment using smartphone-captured color burn wound images

Accurate assessment of burn severity is crucial for the management of burn injuries. Currently, clinicians mainly rely on visual inspection to assess burns, characterized by notable inter-observer discrepancies. In this study, we introduce an innovative analysis platform using color burn wound images for automatic burn severity assessment. To do this, we propose a novel joint-task deep learning model, which is capable of simultaneously segmenting both burn regions and body parts, the two crucial components in calculating the percentage of total body surface area (%TBSA). Asymmetric attention mechanism is introduced, allowing attention guidance from the body part segmentation task to the burn region segmentation task. A user-friendly mobile application is developed to facilitate a fast assessment of burn severity at clinical settings. The proposed framework was evaluated on a dataset comprising 1340 color burn wound images captured on-site at clinical settings. The average Dice coefficients for burn depth segmentation and body part segmentation are 85.12 % and 85.36 %, respectively. The R2 for %TBSA assessment is 0.9136. The source codes for the joint-task framework and the application are released on Github (https://github.com/xjtu-mia/BurnAnalysis). The proposed platform holds the potential to be widely used at clinical settings to facilitate a fast and precise burn assessment.

Improved YOLOv8 for Gas-Flame State Recognition under Low-Pressure Conditions

This paper introduces a lightweight flame detection algorithm, enhancing the accuracy and speed of gas-flame state recognition in low-pressure environments using an improved YOLOv8n model. This method effectively resolves the aforementioned problems. Firstly, GhostNet is integrated into the backbone to form the GhostConv module, reducing the model’s computational parameters. Secondly, the C2f module is improved by integrating RepGhost, forming the C2f_RepGhost module, which performs deep convolution, extends feature dimensions, and simplifies the inference structure. Additionally, the CBAM attention mechanism is added to enhance the model’s ability to capture fine-grained features of flames in both channel and spatial dimensions. The replacement of CIoU with WIoU improves the sensitivity and accuracy of the model’s regression loss. Experimental results on a simulated dataset of the theoretical testbed indicate that compared to the original model, the proposed improvements achieve good performance in low-pressure flame state detection. The model’s parameter count is reduced by 12.64%, the total floating-point operations are reduced by 12.2%, and the detection accuracy is improved by 21.2%. Although the detection frame rate slightly decreases, it still meets real-time detection requirements. The experimental results demonstrate that the feasibility and effectiveness of the proposed algorithm have been significantly improved.

A Fast and Robust Range Alignment Method for ISAR Imaging Based on a Deep Learning Network and Regional Multi-Scale Minimum Entropy Method

In inverse synthetic aperture radar (ISAR) imaging, range alignment (RA) is crucial for translational compensation. To address the need for rapidity and accuracy in the RA process, a fast and robust range alignment method is proposed based on a deep learning network and minimum entropy (ME) method. The proposed method consists primarily of two components: the CNN-RNN attention mechanism network (CRAN) architecture and the regional multi-scale minimum entropy (RMSME) method. The main distinction of this method from existing approaches lies in its utilization of a deep learning network for rapid coarse alignment, followed by the search for minimum entropy within local regions at multiple scales. The integration strategy effectively addresses the current challenges of poor generalization in deep learning networks and low efficiency in the traditional ME method. The experimental results of simulation data indicate that the proposed method achieves the best range alignment performance compared to RNN, CRAN, and the traditional ME method. The experimental results of the measured data further validate the practicality of the proposed method. This research provides reference significance for the joint application of deep learning and traditional methods in the RA process.

A novel metastatic tumor segmentation method with a new evaluation metric in clinic study.

Brain metastases are the most common brain malignancies. Automatic detection and segmentation of brain metastases provide significant assistance for radiologists in discovering the location of the lesion and making accurate clinical decisions on brain tumor type for precise treatment. However, due to the small size of the brain metastases, existing brain metastases segmentation produces unsatisfactory results and has not been evaluated on clinic datasets. In this work, we propose a new metastasis segmentation method DRAU-Net, which integrates a new attention mechanism multi-branch weighted attention module and DResConv module, making the extraction of tumor boundaries more complete. To enhance the evaluation of both the segmentation quality and the number of targets, we propose a novel medical image segmentation evaluation metric: multi-objective segmentation integrity metric, which effectively improves the evaluation results on multiple brain metastases with small size. Experimental results evaluated on the BraTS2023 dataset and collected clinical data show that the proposed method has achieved excellent performance with an average dice coefficient of 0.6858 and multi-objective segmentation integrity metric of 0.5582. Compared with other methods, our proposed method achieved the best performance in the task of segmenting metastatic tumors.

Remote sensing image classification based on non-linear enhanced attention mechanism

Abstract In recent years, Transformer technology has gradually shown promising applications in the field of computer vision, becoming a research hotspot. However, traditional vision Transformers suffer from significant computational burdens. Although previous studies have attempted to alleviate this issue by reducing the computational load of attention mechanisms, their methods still require improvement. Additionally, past research often overlooked the non-linear representation of values within attention mechanisms, posing another challenge to address. Therefore, this paper proposes a novel attention mechanism aimed at reducing the computational burden of traditional attention mechanisms while enhancing their ability to express non-linearities in values. Furthermore, conventional feedforward neural networks typically output results directly after passing through an activation function, which may limit the model’s representational capacity due to the simplistic structure leading to singular feature information. To address this issue, this study introduces a more robust feature fusion convolutional feedforward network. Through these methods, the aim is to enhance the accuracy of Transformers in the field of remote-sensing image classification.

Intelligent fatigue damage tracking and prognostics of composite structures utilizing raw images via interpretable deep learning

In recent years, prognostics gained attention in various industries by optimizing maintenance, boosting operational efficiency, and preventing costly downtime. Central to prognostics is the Remaining Useful Life (RUL), representing the critical time before system failure. Deep learning advancements facilitate RUL forecasting by extracting features from diverse data formats such as time series, images, or sequences thereof, in one, two, or three dimensions, respectively. Yet, predicting RUL from image sequences often relies heavily on resource-intensive techniques like digital image correlation, complicating data acquisition. To address challenges with high-dimensional data and unreliable models, this study introduces ISTRUST, an innovative Transformer-based architecture. ISTRUST (Interpretable Spatiotemporal TRansformer for Understanding STructures) tackles the dual challenges posed by high-dimensional data and the black-box nature of existing models. Leveraging Transformers’ attention mechanism, ISTRUST breaks down the spatiotemporal domain, effectively realizing interpretable RUL predictions under uncertainty using only sparse raw image sequences as input. Evaluated on fatigue-loaded composite samples showcasing crack propagation, ISTRUST interprets the relation between cracks and RUL via the attention mechanism. The results substantiate its capacity to interpret and clarify instances in which predictions may exhibit variability in accuracy. Through the attention mechanism, a strong correlation between the model’s spatiotemporal focus and the RUL predictions is established, making it, to the best of our knowledge, the first model to provide interpretable stochastic RUL predictions directly from sequential images of this nature.

MultiNet 2.0: A lightweight attention-based deep learning network for stenosis measurement in carotid ultrasound scans and cardiovascular risk assessment

BackgroundCardiovascular diseases (CVD) cause 19 million fatalities each year and cost nations billions of dollars. Surrogate biomarkers are established methods for CVD risk stratification; however, manual inspection is costly, cumbersome, and error-prone. The contemporary artificial intelligence (AI) tools for segmentation and risk prediction, including older deep learning (DL) networks employ simple merge connections which may result in semantic loss of information and hence low in accuracy. MethodologyWe hypothesize that DL networks enhanced with attention mechanisms can do better segmentation than older DL models. The attention mechanism can concentrate on relevant features aiding the model in better understanding and interpreting images. This study proposes MultiNet 2.0 (AtheroPoint, Roseville, CA, USA), two attention networks have been used to segment the lumen from common carotid artery (CCA) ultrasound images and predict CVD risks. ResultsThe database consisted of 407 ultrasound CCA images of both the left and right sides taken from 204 patients. Two experts were hired to delineate borders on the 407 images, generating two ground truths (GT1 and GT2). The results were far better than contemporary models. The lumen dimension (LD) error for GT1 and GT2 were 0.13±0.08 and 0.16±0.07 mm, respectively, the best in market. The AUC for low, moderate and high-risk patients’ detection from stenosis data for GT1 were 0.88, 0.98, and 1.00 respectively. Similarly, for GT2, the AUC values for low, moderate, and high-risk patient detection were 0.93, 0.97, and 1.00, respectively.The system can be fully adopted for clinical practice in AtheroEdge™ model by AtheroPoint, Roseville, CA, USA.

Knowledge Representation Learning Method Based on Semantic Enhancement of External Information

Background: Knowledge representation learning aims at mapping entity and relational data in knowledge graphs to a low-dimensional space in the form of vectors. The existing work has mainly focused on structured information representation of triples or introducing only one additional kind of information, which has large limitations and reduces the representation efficiency. Objective: This study aims to combine entity description information and textual relationship description information with triadic structure information, and then use the linear mapping method to linearly transform the structure vector and text vector to obtain the joint representation vector. Methods: A knowledge representation learning (DRKRL) model that fuses external information for semantic enhancement is proposed, which combines entity descriptions and textual relations with a triadic structure. For entity descriptions, a vector representation is performed using a bi-directional long- and short-term memory network (Bi-LSTM) model and an attention mechanism. For the textual relations, a convolutional neural network is used to vectorially encode the relations between entities, and then an attention mechanism is used to obtain valuable information as complementary information to the triad. Results: Link prediction and triadic group classification experiments were conducted on the FB15K, FB15K-237, WN18, WN18RR, and NELL-995 datasets. Theoretical analysis and experimental results show that the DRKRL model proposed in this paper has higher accuracy and efficiency compared with existing models. Conclusion: Combining entity description information and textual relationship description information with triadic structure information can make the model have better performance and effectively improve the knowledge representation learning ability.

Research on English Translation Optimization Algorithm Based on Statistical Machine Learning

In the study titled Research on English Translation Optimization Algorithm Based on Statistical Machine Learning: IAAM-NN (Integrating Advanced Attention Mechanisms with Neural Networks), we explore the fusion of advanced attention mechanisms with neural networks to enhance English translation accuracy. This research delves into the intersection of statistical machine learning and language processing, presenting a novel approach termed IAAM-NN. This method capitalizes on the strengths of neural networks in learning complex patterns and the refined attention mechanisms’ ability to accurately map contextual relationships within text. The core objective is to address the challenges faced in traditional translation algorithms – primarily context misinterpretation and semantic inaccuracies. By harnessing the power of advanced attention mechanisms, the IAAM-NN algorithm effectively deciphers nuanced linguistic structures, ensuring more accurate and contextually relevant translation outputs. This study demonstrates the potential of combining neural network models with enhanced attention processes, illustrating significant improvements in translation quality compared to standard machine learning approaches. The implementation of IAAM-NN marks a step forward in the realm of machine translation, offering insights into developing more sophisticated and reliable translation tools in the future.

A novel heat load prediction model of district heating system based on hybrid whale optimization algorithm (WOA) and CNN-LSTM with attention mechanism

Machine learning models, particularly long short-term memory (LSTM) networks, have been extensively employed for heat load prediction in district heating systems (DHS). Nevertheless, the over-reliance on default hyperparameter settings in most methods hinders further enhancement of prediction accuracy. A novel load prediction model is presented, which integrates the whale optimization algorithm (WOA) to refine the hyperparameters of an LSTM model bolstered by an attention mechanism (ATT) and convolutional neural network (CNN). Three hybrid models (WOA-CNN-ATT-LSTM, PSO-CNN-ATT-LSTM and GA-CNN-ATT-LSTM) are constructed by comparing WOA with particle swarm optimization (PSO) and genetic algorithm (GA). The proposed hybrid models are evaluated against traditional LSTM models using an 1100-hour dataset from a real DHS. The outcomes reveal that the WOA-CNN-ATT-LSTM model surpasses both the PSO-CNN-ATT-LSTM and GA-CNN-ATT-LSTM models in heat load prediction accuracy, achieving improvements of 1.9% and 3.2% respectively, and attaining the highest prediction accuracy (R2=0.9962, MSE=0.0001, MAE=0.0082). Moreover, the WOA-CNN-ATT-LSTM model demonstrates superior performance across various time scales (half-day, one-day, three-days, and one-week), highlighting its robustness in heat load prediction. This novel model adaptively adjusts its hyperparameters to identify the optimal configuration, thereby significantly augmenting the overall predictive capabilities of the model.

IACP-DFSRA: Identification of Anticancer Peptides Based on a Dual-channel Fusion Strategy of ResCNN and Attention

Anticancer peptides (ACPs) have been widely applied in the treatment of cancer owing to good safety, rational side effects, and high selectivity. However, the number of ACPs that have been experimentally validated is limited as identification of ACPs is extremely expensive. Hence, accurate and cost-effective identification methods for ACPs are urgently needed. In this work, we proposed a deep learning-based model, named iACP-DFSRA, for ACPs identification. Specifically, we adopted two kinds of sequence embedding technologies, ProtBert_BFD pre-training language model and handcrafted features to encode protein sequences. Then, the LightGBM was used for feature selection, and the selected features were input into ResCNN and Attention mechanism, respectively, to extract local and global features. Finally, the concatenate features were deeply fused by using the Attention mechanism to allow key features to be paid more attention to by the model and make predictions by fully connected layer. The results of 10-fold cross-validation demonstrated that the iACP-DFSRA model delivered improved results in most metrics with Sp of 94.15%, Sn of 95.32%, Acc of 94.74% and MCC of 89.48% compared to the latest AACFlow model. Indeed, the iACP-DFSRA model is the only model with Acc > 90% and MCC > 80% on this independent test dataset. Furthermore, we have further demonstrated the superiority of our model on additional datasets. In addition, t-SNE and SHAP interpretation analysis demonstrated that it is crucial to use two channels for feature extraction and use the Attention mechanism for deep fusion, which helps the iACP-DFSRA to predict ACPs more effectively.

Multi-scale attention in attention neural network for single image deblurring

Image deblurring, which eliminates blurring artifacts to recover details from a given input image, represents an important task for the computer vision field. Recently, the attention mechanism with deep neural networks (DNN) demonstrates promising performance of image deblurring. However, they have difficulty learning complex blurry and sharp relationships through a balance of spatial detail and high-level contextualized information. Moreover, most existing attention-based DNN methods fail to selectively exploit the information from attention and non-attention branches. To address these challenges, we propose a new approach called Multi-Scale Attention in Attention (MSAiA) for image deblurring. MSAiA incorporates dynamic weight generation by leveraging the joint dependencies of channel and spatial information, allowing for adaptive changes to the weight values in attention and non-attention branches. In contrast to existing attention mechanisms that primarily consider channel or spatial dependencies and do not adequately utilize the information from attention and non-attention branches, our proposed AiA design combines channel-spatial attention. This attention mechanism effectively utilizes the dependencies between channel-spatial information to allocate weight values for attention and non-attention branches, enabling the full utilization of information from both branches. Consequently, the attention branch can more effectively incorporate useful information, while the non-attention branch avoids less useful information. Additionally, we employ a novel multi-scale neural network that aims to learn the relationships between blurring artifacts and the original sharp image by further exploiting multi-scale information. The experimental results prove that the proposed MSAiA achieves superior deblurring performance compared with the state-of-the-art methods.

Visualizing and Understanding Patch Interactions in Vision Transformer.

Vision transformer (ViT) has become a leading tool in various computer vision tasks, owing to its unique self-attention mechanism that learns visual representations explicitly through cross-patch information interactions. Despite having good success, the literature seldom explores the explainability of ViT, and there is no clear picture of how the attention mechanism with respect to the correlation across comprehensive patches will impact the performance and what is the further potential. In this work, we propose a novel explainable visualization approach to analyze and interpret the crucial attention interactions among patches for ViT. Specifically, we first introduce a quantification indicator to measure the impact of patch interaction and verify such quantification on attention window design and indiscriminative patches removal. Then, we exploit the effective responsive field of each patch in ViT and devise a window-free transformer (WinfT) architecture accordingly. Extensive experiments on ImageNet demonstrate that the exquisitely designed quantitative method is shown able to facilitate ViT model learning, leading the top-1 accuracy by 4.28% at most. More remarkably, the results on downstream fine-grained recognition tasks further validate the generalization of our proposal.

BAMRE: Joint extraction model of Chinese medical entities and relations based on Biaffine transformation with relation attention

Electronic Health Records (EHRs) contain various valuable medical entities and their relationships. Although the extraction of biomedical relationships has achieved good results in the mining of electronic health records and the construction of biomedical knowledge bases, there are still some problems. There may be implied complex associations between entities and relationships in overlapping triplets, and ignoring these interactions may lead to a decrease in the accuracy of entity extraction. To address this issue, a joint extraction model for medical entity relations based on a relation attention mechanism is proposed. The relation extraction module identifies candidate relationships within a sentence. The attention mechanism based on these relationships assigns weights to contextual words in the sentence that are associated with different relationships. Additionally, it extracts the subject and object entities. Under a specific relationship, entity vector representations are utilized to construct a global entity matching matrix based on Biaffine transformations. This matrix is designed to enhance the semantic dependencies and relational representations between entities, enabling triplet extraction. This allows the two subtasks of named entity recognition and relation extraction to be interrelated, fully utilizing contextual information within the sentence, and effectively addresses the issue of overlapping triplets.Experimental observations from the CMeIE Chinese medical relation extraction dataset and the Baidu2019 Chinese dataset confirm that our approach yields the superior F1 score across all cutting-edge baselines. Moreover, it offers substantial performance improvements in intricate situations involving diverse overlapping patterns, multitudes of triplets, and cross-sentence triplets.

Class‐wised domain decoupling‐guided adversarial feature learning for cross‐age face recognition

AbstractHow to extract age‐independent identity features from face images has long been the major challenge in cross‐age face recognition task. In this letter, a class‐wised domain decoupling‐guided adversarial feature learning network to extract age‐independent face features is proposed. In this model, an adversarial feature learning module is designed to decompose identity features and age features. Considering that artificially designed age interference suppression strategies are difficult to fit the complex relationship between identity features and age features, an adversarial training strategy incorporated with an attention mechanism in the adversarial feature learning module is introduced to suppress age interference adaptively. Besides, under the idea of maximizing the distribution difference of identity feature domain and age feature domain, a class‐wised domain decoupling module is further designed to guide the model to extract age‐independent identity features. The proposed model has been validated on the well‐known public datasets CALFW and CACD‐VS, where it achieved remarkable recognition accuracy rates of 94.5% and 99.5%, respectively. This represents improvements of 4.2% and 0.1% over the state‐of‐the‐art single‐task‐based models, clearly showcasing the effectiveness of the proposed class‐wised domain decoupling‐guided adversarial feature learning model.

Hypergraph Convolutional Networks with Multi-Ordering Relations for Cross-Document Event Coreference Resolution

Recognizing the coreference relationship between different event mentions in the text (i.e., event coreference resolution) is an important task in natural language processing. It helps to understand the association between various events in the text, and plays an important role in information extraction, question answering systems, and reading comprehension. Existing research has made progress in improving the performance of event coreference resolution, but there are also some shortcomings. For example, most of the existing methods analyze the event data in the document in a serial processing mode, without considering the complex relationship between events, and it is difficult to mine the deep semantics of events. To solve these problems, this paper proposes a cross-document event co-reference resolution method (HGCN-ECR) based on hypergraph convolutional neural networks. Firstly, the BiLSTM-CRF model was used to label the semantic role of the events extracted from a number of scientific and technical kinds of literature. According to the labeling results, the trigger words and non-trigger words of the event were determined, and the multi-document event hypergraph was constructed around the event trigger words. Then hypergraph convolutional neural networks are used to learn higher-order semantic information in multi-document event hypergraphs, and multi-head attention mechanisms are introduced to understand the hidden features of different event relationship types by treating each event relationship as a set of separate attention mechanisms. Finally, the feed-forward neural network and the average link clustering method are used to calculate the coreference score of events and complete the coreference event clustering, and the cross-document event coreference resolution is realized. The experimental results show that the cross-document event co-reference resolution method is superior to the baseline model.

A Method for Extracting Power Entity Relationships Based on Hybrid Neural Networks

In response to the challenges of entity relationship extraction in unstructured text, the author proposes a power entity relationship extraction method based on a hybrid neural network. This method aims to overcome the limitations of existing models in accurately representing contextual environment information, thereby improving the accuracy of the extraction model to meet practical application needs. Firstly, a Bidirectional Gated Recurrent Unit (BiGRU) was designed to better capture contextual information in text sequences. This helps the model to better understand the relationships between entities. Secondly, an attention mechanism was adopted to enable the model to automatically focus on sequence features that have a significant impact on relationships. This helps the model to extract entity relationships more accurately in complex text environments. Finally, a segmented convolutional neural network (PCNN) was introduced to further improve the accuracy of relationship extraction by learning the environmental feature information in the adjusted sequence. This enables the model to better understand the contextual relationships between entities. On the publicly available English dataset SemEval2010Task8, this method achieved satisfactory results, achieving an F1 value of 85.62%. These experiments have confirmed the effectiveness of our method, providing new ideas and support for automatic extraction of entity relationships, and are expected to play an important role in the field of information extraction.

A method for measuring carbon emissions from power plants using a CNN-LSTM-Attention model with Bayesian optimization

Accurate measurement of CO2 emissions from coal-fired power plants is crucial for achieving China's “carbon neutrality” goal; however, traditional CO2 emission measurement methods have several limitations, and emerging prediction models rarely utilize real-time monitoring data from power plants at the micro level. To address these challenges, this study presents a novel hybrid convolutional neural network with long short-term memory (CNN-LSTM)-Attention model, leveraging real-time monitoring data to enhance the accuracy of CO2 emission predictions. The proposed model employs a hybrid architecture that leverages the strength of CNN, LSTM, and Attention mechanism. To address the challenges of multiple hyperparameters and difficult selection, the Bayesian optimization algorithm is used to optimize the hyperparameters ensuring optimal performance. Using historical operational data from a 660 MW generator set in Hubei province, the proposed model was evaluated and validated horizontally by using three evaluation metrics. Additionally, the vertical evaluation and validation of the proposed model were done using the operational data from other months. The results indicate that the proposed model achieved the highest performance across every evaluation index for predicting CO2 emissions from coalfired power plants, and the prediction accuracy of the CO2 emission in the future months is also guaranteed.

A Remaining Useful Life Prediction of Turbofan Engines Based on Multi-Scale Temporal Convolutional Networks with Dual Squeeze-Excitation Attention Mechanism

Abstract To better handle temporal data and delve into learning the features of the data, a turbofan engine residual life prediction method is proposed, which integrates a dual-squeeze-excitation attention mechanism with a multi-scale temporal convolutional network. Firstly, utilizing a sliding window, the extracted multi-dimensional sensor features undergo overlapping sampling to enhance the model’s perception of temporal data. Secondly, a hybrid network prediction model based on DSE-MTCN is constructed, employing multi-scale convolutional kernels to expand the receptive field of convolution, assigning different weights to features, and adaptively allocating weights to hidden layer units. Lastly, the DSE-MTCN prediction model is globally optimized using the RAdam algorithm. The results demonstrate that this method effectively enhances the accuracy and generalization ability of the prediction model.