Deep Attention Network Research Articles

Deep supervision feature refinement attention network for medical image segmentation

The improvement of medical technology is closely related to the development of computers. Deep learning methods have become an important means for medical image processing, and their accuracy in processing lesions will have a significant impact on the final diagnosis result. Currently, some traditional algorithms and classical deep learning methods are no longer able to meet the increasing demand for higher accuracy in medical image processing. In order to achieve better performance in handling the edge and detail features of lesions, we create the Deep Supervision Feature Refinement Attention Network (DSFRA-Net) through extensive experiments. In DSFRA-Net, the Depth Feature Attention Block is created to enhance the long-range dependency between pixels in deep neural networks. The Feature Refinement Block is developed to enhance the details in shallow features. The Adaptive Feature Extraction Block is created to strengthen the fusion of semantic information and detail information. A deep supervision mechanism is used to supervise each layer of the feature reconstruction process, feeding back the training in the form of a loss function to optimize the training. DSFRA-Net experiments on four datasets, all of which show better performance than the current mainstream networks. It shows superior capabilities in areas such as feature continuity and detailed feature processing.

Feature Distillation in Deep Attention Network Against Adversarial Examples.

Deep neural networks (DNNs) are easily fooled by adversarial examples. Most existing defense strategies defend against adversarial examples based on full information of whole images. In reality, one possible reason as to why humans are not sensitive to adversarial perturbations is that the human visual mechanism often concentrates on most important regions of images. A deep attention mechanism has been applied in many computer fields and has achieved great success. Attention modules are composed of an attention branch and a trunk branch. The encoder/decoder architecture in the attention branch has potential of compressing adversarial perturbations. In this article, we theoretically prove that attention modules can compress adversarial perturbations by destroying potential linear characteristics of DNNs. Considering the distribution characteristics of adversarial perturbations in different frequency bands, we design and compare three types of attention modules based on frequency decomposition and reorganization to defend against adversarial examples. Moreover, we find that our designed attention modules can obtain high classification accuracies on clean images by locating attention regions more accurately. Experimental results on the CIFAR and ImageNet dataset demonstrate that frequency reorganization in attention modules can not only achieve good robustness to adversarial perturbations, but also obtain comparable, even higher classification, accuracies on clean images. Moreover, our proposed attention modules can be integrated with existing defense strategies as components to further improve adversarial robustness.

Inferable Deep Distilled Attention Network for Diagnosing Multiple Motor Bearing Faults

Bearing, as a vital component in electric powertrains, is increasingly used globally such as in electric vehicle (EV). Their damages and faults may bring huge cost loss to the industry and even threaten personal safety. This paper proposes an inferable deep distilled attention network (IDDAN) method which is a self-attention mechanism and transfer learning-based method to diagnose and classify multiple bearing faults in various motor drive systems efficiently and accurately. Compared with convolutional networks, the self-attention-based network can better extract the global feature information and easier to benefit from large amounts of pre-training data. Its significance is to accurately classify various faults of the target machine when the labeled data of the target machine is not enough to directly train the diagnosis model. Firstly, this paper attempt to apply the self-attention-based network to build an advanced fault diagnosis model. Secondly, this paper optimizes the structure of networks through knowledge distillation (KD) technique to require a lighter and fast model. Thirdly, this paper proposes a new data augmentation strategy for 1-D vibration signals to provide large-scale pre-training samples for IDDAN. Experiments show that the self-attention mechanism-based model is more likely to benefit from large-scale data. After testing, compared with many methods and other exist similar methods, the proposed method achieves higher classification accuracy and better performance.

Intelligent Recommendation System Based on the Infusion Algorithms with Deep Learning, Attention Network and Clustering

The creation and use of big data have driven the intelligent development of e-commerce. The information generated in e-commerce provides a good means to analyze the behavior of users. How to use this information to give customer recommendations, improve the accuracy of recommendations and protect information security is a topic worth studying. For improving the accuracy of recommendations, analysis of users and tagging of resources are key. The current popular session recommendation algorithms face many problems, such as user interest drift which is difficult to be handled by these algorithms, thus affecting the recommendation accuracy. Based on these problems, this paper proposes a recommendation model based on deep learning, applies it to the clustering analysis of user tagging system, and designs a personalized recommendation algorithm for the tagging system. The model proposed in this paper can effectively analyze not only the interests exhibited by users in the current session, but also their potential long-term interests. By comparing the different performances of different datasets, the experimental results of this paper show that the proposed algorithmic model in this paper helps to dig the interests of different users, thus improving the quality of the recommendation system.

Learning Representative Features by Deep Attention Network for 3D Point Cloud Registration.

Three-dimensional point cloud registration, which aims to find the transformation that best aligns two point clouds, is a widely studied problem in computer vision with a wide spectrum of applications, such as underground mining. Many learning-based approaches have been developed and have demonstrated their effectiveness for point cloud registration. Particularly, attention-based models have achieved outstanding performance due to the extra contextual information captured by attention mechanisms. To avoid the high computation cost brought by attention mechanisms, an encoder-decoder framework is often employed to hierarchically extract the features where the attention module is only applied in the middle. This leads to the compromised effectiveness of the attention module. To tackle this issue, we propose a novel model with the attention layers embedded in both the encoder and decoder stages. In our model, the self-attentional layers are applied in the encoder to consider the relationship between points inside each point cloud, while the decoder utilizes cross-attentional layers to enrich features with contextual information. Extensive experiments conducted on public datasets prove that our model is able to achieve quality results on a registration task.

State of Health prediction of lithium-ion batteries based on temporal degeneration feature extraction with Deep Cycle Attention Network

The State of Health (SOH) forecasting is essential for applying lithium-ion batteries in energy storage systems. The streaming sensor data collected by battery management systems (BMS) contain information about the battery SOH. Therefore, the battery degradation can be treated as the attention-based similarity in the battery cycles from the data point of view. The paper develops the Deep Cycle Attention Network (DCAN) with the attention mechanism to measure the battery state similarity in different cycles with the temporal patterns extracted from the streaming sensor data. The DCAN model can efficiently predict battery SOH under various operational working conditions with the attention mechanism. Essentially, the attention-based similarity regarding the first cycle can describe the battery SOH development in the charge and discharge cycles by extracting the degeneration-related temporal patterns from the streaming sensor data. Based on the MIT-Stanford dataset and NASA PCoE dataset, the prediction performance of the DCAN model is tested under constant current load profiles. Furthermore, the Oxford dataset is used to test the prediction performance with the characterization load profiles under the conditions of driving cycles. As a result, the proposed DCAN model demonstrates high accuracy and strong robustness for the two kinds of load profiles.

Simulating airflows with a deep attention network for physical interpretability

Machine learning model offers interpretability, accuracy, and computational efficiency for flow field calculations

Transformer-based deep reverse attention network for multi-sensory human activity recognition

In today’s era, one of the important applications of Artificial Intelligence (AI) is Human Activity Recognition (HAR). It has a wide range of applicability in health monitoring for patients with chronic diseases, gaming consoles for gesture recognition, etc. Sensor-based HAR systems use signals collected over a period of time to label an activity. When we design an efficient sensor-based HAR system, a model requires learning an optimal association of spatial and temporal features. In this article, we propose a sensor-based HAR technique using the deep learning approach. We present a deep reverse transformer-based attention mechanism to guide the side residual features Unlike the conventional bottom-up approaches for feature fusion, we exploit a top-down feature fusion approach. The reverse attention is self-calibrated throughout the course of learning, which regularizes the attention modules and dynamically adjusts the learning rate. The overall framework outperforms several state-of-the-art methods and is shown to be statistically significant against these methods on five publicly available sensor-based HAR datasets, namely, MHEALTH, USC-HAD, WHARF, UTD-MHAD1, and UTD-MHAD2. Further, we conduct an ablation study to showcase the importance of each of the components of the proposed framework. Source code of this work is available at https://github.com/rishavpramanik/RevTransformerAttentionHAR.

Fast aerodynamics prediction of laminar airfoils based on deep attention network

The traditional method for obtaining aerodynamic parameters of airfoils by solving Navier–Stokes equations is a time-consuming computing task. In this article, a novel data-driven deep attention network (DAN) is proposed for reconstruction of incompressible steady flow fields around airfoils. To extract the geometric representation of the input airfoils, the grayscale image of the airfoil is divided into a set of patches, and these are input into the transformer encoder by embedding. The geometric parameters extracted from the transformer encoder, together with the Reynolds number, angle of attack, flow field coordinates, and distance field, are input into a multilayer perceptron to predict the flow field of the airfoil. Through analysis of a large number of qualitative and quantitative experimental results, it is concluded that the proposed DAN can improve the interpretability of the model while obtaining good prediction accuracy and generalization capability for different airfoils and flow-field states.

Graph-Based Domain Adaptation Few-Shot Learning for Hyperspectral Image Classification

Due to a lack of labeled samples, deep learning methods generally tend to have poor classification performance in practical applications. Few-shot learning (FSL), as an emerging learning paradigm, has been widely utilized in hyperspectral image (HSI) classification with limited labeled samples. However, the existing FSL methods generally ignore the domain shift problem in cross-domain scenes and rarely explore the associations between samples in the source and target domain. To tackle the above issues, a graph-based domain adaptation FSL (GDAFSL) method is proposed for HSI classification with limited training samples, which utilizes the graph method to guide the domain adaptation learning process in a uniformed framework. First, a novel deep residual hybrid attention network (DRHAN) is designed to extract discriminative embedded features efficiently for few-shot HSI classification. Then, a graph-based domain adaptation network (GDAN), which combines graph construction with domain adversarial strategy, is proposed to fully explore the domain correlation between source and target embedded features. By utilizing the fully explored domain correlations to guide the domain adaptation process, a domain invariant feature metric space is learned for few-shot HSI classification. Comprehensive experimental results conducted on three public HSI datasets demonstrate that GDAFSL is superior to the state-of-the-art with a small sample size.

Explainable Deep Hypergraph Learning Modeling the Peptide Secondary Structure Prediction.

Accurately predicting peptide secondary structures remains a challenging task due to the lack of discriminative information in short peptides. In this study, PHAT is proposed, a deep hypergraph learning framework for the prediction of peptide secondary structures and the exploration of downstream tasks. The framework includes a novel interpretable deep hypergraph multi-head attention network that uses residue-based reasoning for structure prediction. The algorithm can incorporate sequential semantic information from large-scale biological corpus and structural semantic information from multi-scale structural segmentation, leading to better accuracy and interpretability even with extremely short peptides. The interpretable models are able to highlight the reasoning of structural feature representations and the classification of secondary substructures. The importance of secondary structures in peptide tertiary structure reconstruction and downstream functional analysis is further demonstrated, highlighting the versatility of our models. To facilitate the use of the model, an online server is established which is accessible via http://inner.wei-group.net/PHAT/. The work is expected to assist in the design of functional peptides and contribute to the advancement of structural biology research.

Interpretable bilinear attention network with domain adaptation improves drug–target prediction

Predicting drug–target interaction is key for drug discovery. Recent deep learning-based methods show promising performance, but two challenges remain: how to explicitly model and learn local interactions between drugs and targets for better prediction and interpretation and how to optimize generalization performance of predictions on novel drug–target pairs. Here, we present DrugBAN, a deep bilinear attention network (BAN) framework with domain adaptation to explicitly learn pairwise local interactions between drugs and targets, and adapt in response to out-of-distribution data. DrugBAN works on drug molecular graphs and target protein sequences to perform prediction, with conditional domain adversarial learning to align learned interaction representations across different distributions for better generalization on novel drug–target pairs. Experiments on three benchmark datasets under both in-domain and cross-domain settings show that DrugBAN achieves the best overall performance against five state-of-the-art baseline models. Moreover, visualizing the learned bilinear attention map provides interpretable insights from prediction results.

Deep Embedding-Attention-Refinement for Sparse-View CT Reconstruction

Tomographic image reconstruction with deep learning is an emerging field of applied artificial intelligence. Reducing radiation dose with sparse views’ reconstruction is a significant task in cardiac imaging. Many efforts are contributing to sparse-view tomography imaging, but it is still a challenge for achieving good images from high sparse-view level, such as 60 views. In this study, we proposed a Deep Embedding-Attention-Refinement (DEAR) network to fundamentally address this challenge. DEAR consists of three modules including deep embedding, deep attention, and deep refinement. The measurement is extended by deep embedding network to generate artifact-reduction images. Then, the deep attention network is employed to remove sparse-view artifacts and correct wrong details introduced by deep embedding network. Finally, the deep refinement module is used to refine finer image features and structures. The results on clinical datasets demonstrate the efficiency of our proposed DEAR in edge preservation and feature recovery.

Deep Residual Weight-Sharing Attention Network With Low-Rank Attention for Visual Question Answering

The attention-based networks have become prevailing recently in visual question answering (VQA) due to their high performances. However, the extensive memory consumption of attention-based models poses excessive-high demand for the implementation equipment, raising concerns about their future application scenarios. Therefore, designing an efficient and lightweight VQA model is central to expanding possible application areas. Our work presents a novel lightweight attention-based VQA model, namely residual weight-sharing attention network (RWSAN), consisting of residual weight-sharing attention (RWSA) layers cascaded in depth. Each RWSA layer models the textual representation with self residual weight-sharing attention (SRWSA) and captures question features and question-image interactions with self-guided residual weight-sharing attention (SGRWSA). Inside each RWSA layer, the proposed low-rank attention units perform residual learning with learned connection patterns and shared parameters, and every stacked RWSA layer also uses the same parameters. Extensive ablation experiments with quantitative and qualitative analysis are conducted to illustrate the effectiveness and generality of RWSA. Experiments on VQA-v2, GQA, and CLEVR datasets show that the RWSAN achieves competitive performance with much fewer parameters over the state-of-the-art methods.

DMANet_KF: Tropical Cyclone Intensity Estimation Based on Deep Learning and Kalman Filter From Multispectral Infrared Images

It is very crucial to identify the intensity of tropical cyclone (TC) accurately. In this article, a novel TC intensity estimation method is proposed to estimate the TC intensity from multispectral infrared images in the Northwest Pacific Basin. A deep multisource attention network (DMANet) is proposed to model the dynamics of multispectral infrared images along the spatial dimension. We first introduce a message-passing enhancement module based on the conditional random fields to process multispectral infrared images. Multispectral data transfer the complementary information to refine the features of TC. Second, we utilize a local global attention module to make the model focus on local key features (i.e., the typhoon eye) and obtain deeper global semantic information of TC. The ablation experiment is set up in the same dataset and computing environment to verify the effectiveness of each module. Finally, we use a Kalman filter to correct the error of TC intensity during its lifetime estimated by the DMANet model. After using Kalman filter, the evolution of TC intensity becomes smooth and corresponding root-mean-square error (RMSE) decreases from 9.79 to 7.82 knots. Compared with the best result of the existing TC intensity estimation method, the RMSE of our method is reduced by 9.07%. Therefore, the proposed TC intensity estimation method shows a great potential for accurately estimating the TC intensity.

Gesture recognition of continuous wavelet transform and deep convolution attention network.

To solve the problem of missing data features using a deep convolutional neural network (DCNN), this paper proposes an improved gesture recognition method. The method first extracts the time-frequency spectrogram of surface electromyography (sEMG) using the continuous wavelet transform. Then, the Spatial Attention Module (SAM) is introduced to construct the DCNN-SAM model. The residual module is embedded to improve the feature representation of relevant regions, and reduces the problem of missing features. Finally, experiments with 10 different gestures are done for verification. The results validate that the recognition accuracy of the improved method is 96.1%. Compared with the DCNN, the accuracy is improved by about 6 percentage points.

DeepTPA-Net: A Deep Triple Attention Network for sEMG-Based Hand Gesture Recognition

DeepTPA-Net: A Deep Triple Attention Network for sEMG-Based Hand Gesture Recognition

A Novel Spectral Indices-Driven Spectral-Spatial-Context Attention Network for Automatic Cloud Detection

Cloud detection is a fundamental step for optical satellite image applications. Although existing deep learning method can provide more accurate cloud detection result. However, performance of these methods rely on a large number of label samples, whose collection is time-consuming and high cost. In addition, cloud detection is challenging in high brightness scenes due to due to cloud and high brightness object have a similar spectral feature. In this study, we propose a cloud index driven spectral-spatial-context attention network (SSCA-net) for cloud detection, which relies on no effort to manually collect label samples and can improve the accuracy cloud detection in high brightness scenes. The label samples are automatically generated from cloud index by using dual-threshold, which is then expanded to improve the completeness of cloud mask labels. We designed SSCA-net with the spectral-spatial-context aware module and spectral-spatial-context information aggregation module, aimed to improve the accuracy cloud detection in high brightness scenes. The results show that the proposed SSCA-net achieved good performance with average overall accuracy of 97.69% and average Kappa coefficient of 92.71% on Sentinel-2 and Landsat-8 dataset. This paper provides a fresh insight into how advanced deep attention network and cloud index can be integrated to obtain high accuracy of cloud detection on high brightness scenes.

In-field classification of the asymptomatic biotrophic phase of potato late blight based on deep learning and proximal hyperspectral imaging

Effective detection of potato late blight (PLB) is an essential aspect of potato cultivation. However, it is a challenge to detect late blight in asymptomatic biotrophic phase in fields with conventional imaging approaches because of the lack of visual symptoms in the canopy. Hyperspectral imaging can capture spectral signals from a wide range of wavelengths also outside the visual wavelengths. Here, we propose a deep learning classification architecture for hyperspectral images by combining 2D convolutional neural network (2D-CNN) and 3D-CNN with deep cooperative attention networks (PLB-2D-3D-A). First, 2D-CNN and 3D-CNN are used to extract rich spectral space features, and then the attention mechanism AttentionBlock and SE-ResNet are used to emphasize the salient features in the feature maps and increase the generalization ability of the model. The dataset is built with 15,360 images (64x64x204), cropped from 240 raw images captured in an experimental field with over 20 potato genotypes. The accuracy in the test dataset of 2000 images reached 0.739 in the full band and 0.790 in the specific bands (492 nm, 519 nm, 560 nm, 592 nm, 717 nm and 765 nm). This study shows an encouraging result for classification of the asymptomatic biotrophic phase of PLB disease with deep learning and proximal hyperspectral imaging.

DARI-Mark: Deep Learning and Attention Network for Robust Image Watermarking

At present, deep learning has achieved excellent achievements in image processing and computer vision and is widely used in the field of watermarking. Attention mechanism, as the research hot spot of deep learning, has not yet been applied in the field of watermarking. In this paper, we propose a deep learning and attention network for robust image watermarking (DARI-Mark). The framework includes four parts: an attention network, a watermark embedding network, a watermark extraction network, and an attack layer. The attention network used in this paper is the channel and spatial attention network, which calculates attention weights along two dimensions, channel and spatial, respectively, assigns different weights to pixels in different channels at different positions and is applied in the watermark embedding and watermark extraction stages. Through end-to-end training, the attention network can locate nonsignificant areas that are insensitive to the human eye and assign greater weights during watermark embedding, and the watermark embedding network selects this region to embed the watermark and improve the imperceptibility. In watermark extraction, by setting the loss function, larger weights can be assigned to watermark-containing features and small weights to noisy signals, so that the watermark extraction network focuses on features about the watermark and suppresses noisy signals in the attacked image to improve robustness. To avoid the phenomenon of gradient disappearance or explosion when the network is deep, both the embedding network and the extraction network have added residual modules. Experiments show that DARI-Mark can embed the watermark without affecting human subjective perception and that it has good robustness. Compared with other state-of-the-art watermarking methods, the proposed framework is more robust to JPEG compression, sharpening, cropping, and noise attacks.