Deep Attention Network Research Articles

Adversarial imitation learning with deep attention network for swarm systems

AbstractSwarm systems consist of a large number of interacting individuals, which exhibit complex behavior despite having simple interaction rules. However, crafting individual motion policies that can manifest desired collective behaviors poses a significant challenge due to the intricate relationship between individual policies and swarm dynamics. This paper addresses this issue by proposing an imitation learning method, which derives individual policies from collective behavior data. The approach leverages an adversarial imitation learning framework, with a deep attention network serving as the individual policy network. Our method successfully imitates three distinct collective behaviors. Utilizing the ease of analysis provided by the deep attention network, we have verified that the individual policies underlying a certain collective behavior are not unique. Additionally, we have analyzed the different individual policies discovered. Lastly, we validate the applicability of the proposed method in designing policies for swarm robots through practical implementation on swarm robots.

Polarity-aware deep attention network for aspect-based sentiment analysis

Polarity-aware deep attention network for aspect-based sentiment analysis

Predicting normalized difference vegetation index using a deep attention network with bidirectional GRU: a hybrid parametric optimization approach

AbstractScalable and accurate normalized difference vegetation index (NDVI) prediction is necessary to track the status of vegetation and the environment and to support proper ecological management. Herein, we present an innovative deep-learning approach to improve NDVI prediction performances by considering enhanced temporal modeling and hybrid optimization processes. The analysis is based on a core model that integrates a Bidirectional Gated Recurrent Unit (BiGRU) with the profound attention feature since the primary research incorporates the capability of complex temporal in addition to NDVI-time series value. The model performs better through a dual algorithm combining the waterwheel plant algorithm (WWPA) and statistical fractal search (SFS) named WWPASFS-BiGRU. The proposed approach is evaluated using real-world NDVI datasets, demonstrating its capability to outperform traditional models and state-of-the-art deep learning methods. Key performance metrics highlight the model’s accuracy, with a root mean square error (RMSE) as low as 0.00011, reflecting its superior predictive ability. Comparative experiments showcase the robustness of our model across different environmental conditions and geographical settings, affirming its applicability in diverse ecological forecasting scenarios. Additionally, extensive statistical validation, including ANOVA and Wilcoxon tests, confirms the model’s consistency and reliability. The effectiveness of the WWPASFS-BiGRU model is illustrated through applications in predicting NDVI trends across regions in Saudi Arabia, providing critical insights for ecosystem management and sustainable development planning.

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

Super-resolution reconstruction improves multishell diffusion: using radiomics to predict adult-type diffuse glioma IDH and grade.

Multishell diffusion scanning is limited by low spatial resolution. We sought to improve the resolution of multishell diffusion images through deep learning-based super-resolution reconstruction (SR) and subsequently develop and validate a prediction model for adult-type diffuse glioma, isocitrate dehydrogenase status and grade 2/3 tumors. A simple diffusion model (DTI) and three advanced diffusion models (DKI, MAP, and NODDI) were constructed based on multishell diffusion scanning. Migration was performed with a generative adversarial network based on deep residual channel attention networks, after which images with 2x and 4x resolution improvements were generated. Radiomic features were used as inputs, and diagnostic models were subsequently constructed via multiple pipelines. This prospective study included 90 instances (median age, 54.5 years; 39 men) diagnosed with adult-type diffuse glioma. Images with both 2x- and 4x-improved resolution were visually superior to the original images, and the 2x-improved images allowed better predictions than did the 4x-improved images (P<.001). A comparison of the areas under the curve among the multiple pipeline-constructed models revealed that the advanced diffusion models did not have greater diagnostic performance than the simple diffusion model (P>.05). The NODDI model constructed with 2x-improved images had the best performance in predicting isocitrate dehydrogenase status (AUC_validation=0.877; Brier score=0.132). The MAP model constructed with the original images performed best in classifying grade 2 and grade 3 tumors (AUC_validation=0.806; Brier score=0.168). SR improves the resolution of multishell diffusion images and has different advantages in achieving different goals and creating different target diffusion models.

DMSCA: deep multiscale cross-modal attention network for hyperspectral and light detection and ranging data fusion and joint classification

DMSCA: deep multiscale cross-modal attention network for hyperspectral and light detection and ranging data fusion and joint classification

Inf-Att-OSVNet: information theory based feature selection and deep attention networks for online signature verification

Inf-Att-OSVNet: information theory based feature selection and deep attention networks for online signature verification

Channel estimation for reconfigurable intelligent surface‐aided millimeter‐wave massive multiple‐input multiple‐output system with deep residual attention network

Channel estimation for reconfigurable intelligent surface‐aided millimeter‐wave massive multiple‐input multiple‐output system with deep residual attention network

Deep attention network for identifying ligand-protein binding sites

One of the critical aspects of structure-based drug design is to choose important druggable binding sites in the protein's crystallography structures. As experimental processes are costly and time-consuming, computational drug design using machine learning algorithms is recommended. Over recent years, deep learning methods have been utilized in a wide variety of research applications such as binding site prediction. In this study, a new combination of attention blocks in the 3D U-Net model based on semantic segmentation methods is used to improve localization of pocket prediction. The attention blocks are tuned to find which point and channel of features should be emphasized along spatial and channel axes. Our model's performance is evaluated through extensive experiments on several datasets from different sources, and the results are compared to the most recent deep learning-based models. The results indicate the proposed attention model (Att-UNet) can predict binding sites accurately, i.e. the overlap of the predicted pocket using the proposed method with the true binding site shows statistically significant improvement when compared to other state-of-the-art models. The attention blocks may help the model focus on the target structure by suppressing features in irrelevant regions.

Hyperspectral image classification based on deep separable residual attention network

Hyperspectral image have rich spatial and spectral information, and how to fully extract and utilize the features of these two dimensions is a research hotspot in hyperspectral classification methods. At present, the unique convolutional operation and deep feature extraction structure of convolutional neural network enable them to have stronger feature representation capabilities and achieve good results in hyperspectral image classification. However, CNN methods do not assign different weights based on the importance of features in the feature extraction process, making it difficult to effectively utilize key features, and most importantly, using fixed shaped convolution kernel can easily overlook the differences between hyperspectral image features. A hyperspectral image classification method based on deep separable residual attention network is proposed to address the above issues. Firstly, to reduce the correlation between hyperspectral image data and minimize the interference of redundant information, principal component analysis is used to reduce the dimensionality of hyperspectral image. Secondly, a shallow feature extraction module is constructed, which can dynamically adjust the size of the receptive field according to the actual situation of the image, adaptively extract shallow features, and reduce the loss of original image features. Then, a depthwise separable residual attention mechanism module is proposed, based on which features are extracted. Starting from global and local features, contextual information on image features in channel and spatial domains is extracted. Finally, use a multi-scale feature fusion module to fully integrate feature maps at different scales. Using Indian Pines, Pavia University and Botswana as experimental datasets, the overall classification accuracy of this paper's method is 98.47 %, 98.70 %, 98.83 % with only 50, 50, 30 training samples per class. The Kappa coefficient is 98.25 %, 98.27 %, and 98.73 %, respectively. Compared with advanced methods, this method not only has higher classification accuracy, but also fully utilizes key features at various network levels.

Precision prediction in grinding processes based on displacement signal conversion into recurrent plot

The processing of grinding data and the prediction of accuracy are extremely complex; this paper proposes a novel prediction method to ensure the machining accuracy of computer numerical control (CNC) grinding machines. It consists of two components, namely the filter decomposition recurrence plot (RP) transformation (FDRP) and the deep inverted residual attention network (DIRAN). A pipeline named FDRP has been designed to address the issues in the processing of data and the shortcomings of RP. Firstly, the displacement signals undergo filtering to preserve essential grinding information while effectively removing noise. Secondly, long-time series signals are decomposed and augmented based on the characteristics of the machining process. Lastly, an RP transformation is applied to one-dimensional time series data, resulting in the generation of images that accurately represent the grinding process. Furthermore, this paper proposes a novel machining accuracy prediction model. The DIRAN uses a multi-layer inverse residual network structure combined with attention mechanism to extract the features of two-dimensional RP, and its performance is better than other typical prediction methods. It can be applied to predict the polar angle of the workpiece in industrial processing and reduce the defect rate.

Fuzzy Attention Neural Network to Tackle Discontinuity in Airway Segmentation.

Airway segmentation is crucial for the examination, diagnosis, and prognosis of lung diseases, while its manual delineation is unduly burdensome. To alleviate this time-consuming and potentially subjective manual procedure, researchers have proposed methods to automatically segment airways from computerized tomography (CT) images. However, some small-sized airway branches (e.g., bronchus and terminal bronchioles) significantly aggravate the difficulty of automatic segmentation by machine learning models. In particular, the variance of voxel values and the severe data imbalance in airway branches make the computational module prone to discontinuous and false-negative predictions, especially for cohorts with different lung diseases. The attention mechanism has shown the capacity to segment complex structures, while fuzzy logic can reduce the uncertainty in feature representations. Therefore, the integration of deep attention networks and fuzzy theory, given by the fuzzy attention layer, should be an escalated solution for better generalization and robustness. This article presents an efficient method for airway segmentation, comprising a novel fuzzy attention neural network (FANN) and a comprehensive loss function to enhance the spatial continuity of airway segmentation. The deep fuzzy set is formulated by a set of voxels in the feature map and a learnable Gaussian membership function. Different from the existing attention mechanism, the proposed channel-specific fuzzy attention addresses the issue of heterogeneous features in different channels. Furthermore, a novel evaluation metric is proposed to assess both the continuity and completeness of airway structures. The efficiency, generalization, and robustness of the proposed method have been proved by training on normal lung disease while testing on datasets of lung cancer, COVID-19, and pulmonary fibrosis.

DCSIAN: A novel deep cross-scale interactive attention network for fault diagnosis of aviation hydraulic pumps and generalizable applications

Channel attention (CA) has been wildly applied to enhance the diagnosis performance of multiscale convolution (MSC)-based diagnosis methods. Nevertheless, most of the existing CA modules only consider the internal local correlation among different channels within each scale feature, but ignore the global correlation among different scales, restricting further improvement. To address this issue, a novel deep cross-scale interactive attention network (DCSIAN) is developed to achieve accurate fault diagnosis for aviation hydraulic pumps under high-noise environments. Specifically, a novel cross-scale interactive attention module (CSIAM) is developed and introduced into MSC to learn complementary and rich multiscale features from original vibration signals. CSIAM adopts two cascaded submodules to focus on local channel correlation and global scale correlation simultaneously. Local channel correlation is used to adaptively measure the importance of different channel feature within each scale, while global scale correlation is used to dynamically determine the contribution of each scale feature to the final diagnostic result. In this way, the fault-related information at different scales can be fully captured and utilized. Finally, the effectiveness of DCSIAN is validated by a series of experimental comparisons on an aviation hydraulic pump dataset and a bearing dataset with various types noise.

Reconstruction of unstable heavy particles using deep symmetry-preserving attention networks

Reconstructing unstable heavy particles requires sophisticated techniques to sift through the large number of possible permutations for assignment of detector objects to the underlying partons. An approach based on a generalized attention mechanism, symmetry preserving attention networks (SPA-NET), has been previously applied to top quark pair decays at the Large Hadron Collider which produce only hadronic jets. Here we extend the SPA-NET architecture to consider multiple input object types, such as leptons, as well as global event features, such as the missing transverse momentum. In addition, we provide regression and classification outputs to supplement the parton assignment. We explore the performance of the extended capability of SPA-NET in the context of semi-leptonic decays of top quark pairs as well as top quark pairs produced in association with a Higgs boson. We find significant improvements in the power of three representative studies: a search for tt¯H\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$t\\bar{t}H$$\\end{document}, a measurement of the top quark mass, and a search for a heavy Z′\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${Z}^{{\\prime} }$$\\end{document} decaying to top quark pairs. We present ablation studies to provide insight on what the network has learned in each case.

Improving multi-step prediction performance of multi-channel QoT over optical backbone networks: deep echo state attention network

Improving multi-step prediction performance of multi-channel QoT over optical backbone networks: deep echo state attention network

DRANetSplicer: A Splice Site Prediction Model Based on Deep Residual Attention Networks.

The precise identification of splice sites is essential for unraveling the structure and function of genes, constituting a pivotal step in the gene annotation process. In this study, we developed a novel deep learning model, DRANetSplicer, that integrates residual learning and attention mechanisms for enhanced accuracy in capturing the intricate features of splice sites. We constructed multiple datasets using the most recent versions of genomic data from three different organisms, Oryza sativa japonica, Arabidopsis thaliana and Homo sapiens. This approach allows us to train models with a richer set of high-quality data. DRANetSplicer outperformed benchmark methods on donor and acceptor splice site datasets, achieving an average accuracy of (96.57%, 95.82%) across the three organisms. Comparative analyses with benchmark methods, including SpliceFinder, Splice2Deep, Deep Splicer, EnsembleSplice, and DNABERT, revealed DRANetSplicer's superior predictive performance, resulting in at least a (4.2%, 11.6%) relative reduction in average error rate. We utilized the DRANetSplicer model trained on O. sativa japonica data to predict splice sites in A. thaliana, achieving accuracies for donor and acceptor sites of (94.89%, 94.25%). These results indicate that DRANetSplicer possesses excellent cross-organism predictive capabilities, with its performance in cross-organism predictions even surpassing that of benchmark methods in non-cross-organism predictions. Cross-organism validation showcased DRANetSplicer's excellence in predicting splice sites across similar organisms, supporting its applicability in gene annotation for understudied organisms. We employed multiple methods to visualize the decision-making process of the model. The visualization results indicate that DRANetSplicer can learn and interpret well-known biological features, further validating its overall performance. Our study systematically examined and confirmed the predictive ability of DRANetSplicer from various levels and perspectives, indicating that its practical application in gene annotation is justified.

Simple and deep graph attention networks

Graph Attention Networks (GATs) and Graph Convolutional Neural Networks (GCNs) are two state-of-the-art architectures in Graph Neural Networks (GNNs). It is well known that both models suffer from performance degradation when more GNN layers are stacked, and many works have been devoted to address this problem. We notice that main research efforts in the line focus on the GCN models, and their techniques cannot well fit the GAT models due to the inherent difference between these two architectures. In GAT, the attention mechanism is limited as it ignores the overwhelming propagation from certain nodes as the number of layers increases. To sufficiently utilize the expressive power of GAT, we propose a new version of GAT named Layer-wise Self-adaptive GAT (LSGAT), which can effectively alleviate the oversmoothing issue in deep GAT and is strictly more expressive than GAT. We redesign the attention coefficients computation mechanism adaptively adjusted by layer depth, which considers both immediate neighboring and non-adjacent nodes from a global view. The experimental evaluation confirms that LSGAT consistently achieves better results on node classification tasks over relevant counterparts.

Adapting face recognition to the masked world: leveraging deep attention networks with a custom dataset

In the era of COVID-19, past face recognition algorithms' performance is downgraded due to the partial occlusion of face mask. A new Indian face image dataset has been proposed titled, Handcrafted Indian Face (HIF) dataset, addressing the issues, viz. variegated illumination, pose, and partial occlusion conditions. It bridges the gap between the performance of the DL models used, pre and post COVID-19 effect. A novel idea for choosing the train-test sample has been presented in the paper, which improves the accuracy on existing state of art DL models. In this paper, a new DL architecture has been proposed named the InceptBlock Enhanced Attention Fusion Network (IBEAFNet) which consists of the combination of ECBAM (Enhanced Convolution Block Attention Module) and InceptionV3 architecture. The proposed architecture's attention layer placement allows it to suppress less relevant mask regions of face, while emphasizing on significant fine and coarse level features with reduced complexity. IBEAFNet is trained and tested on two existing datasets, viz. Casia & Yale (including simulated masked images) and the proposed HIF dataset. The performance of IBEAFNet is compared with the results fetched by changing the attention layers in IBEAFNet with the blocks of SENet and CBAM. IBEAFNet outperformed the state-of-art models with the accuracy of 91.00%, 89.5%, and 93.00% on CASIA, Yale, and HIF dataset, respectively.

MSMVAN: Multi Step Multi Variate Deep Attention Network for Renewable Energy Forecast

MSMVAN: Multi Step Multi Variate Deep Attention Network for Renewable Energy Forecast

G-DGANet: Gated deep graph attention network with reinforcement learning for solving traveling salesman problem

Combinatorial optimization problem (COP) is an NP-hard problem for which finding an optimal solution is difficult, especially as the problem size increases. The Traveling Salesman Problem (TSP), one of the COPs that can be formulated over a graph, is a well-researched area in operations research and computer science. Deep Reinforcement Learning (DRL) is now regarded as a promising approach for solving TSP and other NP hard problems. In this paper, we propose a novel Gated Deep Graph Attention Network (G-DGANet) which builds upon the existing Graph Neural Network (GNN) to solve TSP. The proposed G-DGANet uses gating mechanism between subsequent layers of the network to extract representations of nodes deeper in the network without loss in performance. G-DGANet also designs a novel aggregator to construct global graph embeddings from different embedding preferences. In addition, to effectively learn underlying structure of a graph, G-DGANet integrates node and edge information of the graph while updating node representations in the message passing mechanism. We used proximal policy optimization (PPO) to train G-DGANet on randomly generated instances. We conducted an experiment on randomly generated instances and on real-world road network data generated from digital maps to verify the performance of G-DGANet. The findings from experiments demonstrate that G-DGANet outperforms most traditional heuristics and existing DRL approaches, with high generalization abilities from random instance training to random instance testing and real-world road network instance testing.