Attention Neural Network Research Articles

Multi-layer graph attention neural networks for accurate drug-target interaction mapping

In the crucial process of drug discovery and repurposing, precise prediction of drug-target interactions (DTIs) is paramount. This study introduces a novel DTI prediction approach—Multi-Layer Graph Attention Neural Network (MLGANN), through a groundbreaking computational framework that effectively harnesses multi-source information to enhance prediction accuracy. MLGANN not only strides forward in constructing a multi-layer DTI network by capturing both direct interactions between drugs and targets as well as their multi-level information but also amalgamates Graph Convolutional Networks (GCN) with a self-attention mechanism to comprehensively integrate diverse data sources. This method exhibited significant performance surpassing existing approaches in comparative experiments, underscoring its immense potential in elevating the efficiency and accuracy of DTI predictions. More importantly, this study accentuates the significance of considering multi-source data information and network heterogeneity in the drug discovery process, offering new perspectives and tools for future pharmaceutical research.

Attentive graph neural network models for the prediction of blood brain barrier permeability.

The blood brain barrier's (BBB) unique endothelial cells and tight junctions selectively regulate passage of molecules to the central nervous system (CNS) to prevent pathogen entry and maintain neural homeostasis. Various neurological conditions and neurodegenerative diseases benefit from small molecules capable of BBB penetration (BBBP) to elicit a therapeutic effect. Predicting BBBP often involves in silico assessment of molecular properties such as lipophilicity (log P ) and polar surface area (PSA) using the CNS multiparameter optimization (MPO) method. This study curated an open-source dataset to benchmark rigorously machine learning (ML) and neural network (NN) models with each other and with MPO for predicting BBBP. Our analysis demonstrated that AI models, especially attentive NNs using stereochemical features, significantly outperform MPO in predicting BBBP. An attentive graph neural network (GNN), we refer to as CANDID-CNS™, achieved a 0.23-0.26 higher AUROC score than MPO on full test sets, and a 0.17-0.19 higher score on stereoisomers filtered subsets. Regarding stereoisomers that differ in BBBP, which MPO cannot distinguish, attentive GNNs correctly classify these with AUROC and MCC metrics comparable to or better than MPO's AUROC and MCC on less difficult test molecules. These findings suggest that integrating attentive GNN models into pharmaceutical drug discovery processes can substantially improve prediction rates, and thereby reduce the timeline, cost, and increase probability of success of designing brain penetrant therapeutics for the treatment of a wide variety of neurological and neurodegenerative diseases.

GGAS2SN: Gated Graph and SmilesToSeq Network for Solubility Prediction.

Aqueous solubility is a critical physicochemical property of drug discovery. Solubility is a key issue in pharmaceutical development because it can limit a drug's absorption capacity. Accurate solubility prediction is crucial for pharmacological, environmental, and drug development studies. This research introduces a novel method for solubility prediction by combining gated graph neural networks (GGNNs) and graph attention neural networks (GATs) with Smiles2Seq encoding. Our methodology involves converting chemical compounds into graph structures with nodes representing atoms and edges indicating chemical bonds. These graphs are then processed by using a specialized graph neural network (GNN) architecture. Incorporating attention mechanisms into GNN allows for capturing subtle structural dependencies, fostering improved solubility predictions. Furthermore, we utilized the Smiles2Seq encoding technique to bridge the semantic gap between molecular structures and their textual representations. Smiles2Seq seamlessly converts chemical notations into numeric sequences, facilitating the efficient transfer of information into our model. We demonstrate the efficacy of our approach through comprehensive experiments on benchmark solubility data sets, showcasing superior predictive performance compared to traditional methods. Our model outperforms existing solubility prediction models and provides interpretable insights into the molecular features driving solubility behavior. This research signifies an important advancement in solubility prediction, offering potent tools for drug discovery, formulation development, and environmental assessments. The fusion of GGNN and Smiles2Seq encoding establishes a robust framework for accurately forecasting solubility across various chemical compounds, fostering innovation in various domains reliant on solubility data.

Research on Graph Multi-Attention Neural Network for Power System Transient Stability Assessment

Abstract Diagnosing the stability of power grids based on artificial intelligence technology is a research trend. The existing artificial intelligence stability analysis methods rely on a large number of fault case data, while the graph neural network method ignores the correlation characteristics of nodes themselves and the correlation of long-distance nodes. In order to solve the problems, the graph multi-attention neural network (GMANN) was proposed. The self-attention, which characterizes the correlation between different state quantities of a single node, is proposed, firstly. Then, long-distance association attention, which represents the correlation of distant nodes in the power grid, was proposed. The long-distance correlation attention and the adjacent correlation attention in graph attention network are fused to form a global attention that reflects the global importance of the power grid. The channel attention that characterizes the importance of different pooling methods of graph convolutional network is extracted and used to obtain the importance of different convolution operations. Finally, based on the multi-source attention fusion strategy, global attention and channel attention are embedded in the graph attention neural network to form a multi-level evaluation model to achieve power grid stability evaluation efficiently guided by multiple attentions. The proposed GMANN is verified based on simulated fault cases obtained from the 10-machine 39-node system in New England. The results show that the accuracy of the GMANN can reach 97.34%, which is better than other methods. And the missed judgment rate of important indicators is significantly better than other methods.

Multi-scale attention in attention neural network for single image deblurring

Image deblurring, which eliminates blurring artifacts to recover details from a given input image, represents an important task for the computer vision field. Recently, the attention mechanism with deep neural networks (DNN) demonstrates promising performance of image deblurring. However, they have difficulty learning complex blurry and sharp relationships through a balance of spatial detail and high-level contextualized information. Moreover, most existing attention-based DNN methods fail to selectively exploit the information from attention and non-attention branches. To address these challenges, we propose a new approach called Multi-Scale Attention in Attention (MSAiA) for image deblurring. MSAiA incorporates dynamic weight generation by leveraging the joint dependencies of channel and spatial information, allowing for adaptive changes to the weight values in attention and non-attention branches. In contrast to existing attention mechanisms that primarily consider channel or spatial dependencies and do not adequately utilize the information from attention and non-attention branches, our proposed AiA design combines channel-spatial attention. This attention mechanism effectively utilizes the dependencies between channel-spatial information to allocate weight values for attention and non-attention branches, enabling the full utilization of information from both branches. Consequently, the attention branch can more effectively incorporate useful information, while the non-attention branch avoids less useful information. Additionally, we employ a novel multi-scale neural network that aims to learn the relationships between blurring artifacts and the original sharp image by further exploiting multi-scale information. The experimental results prove that the proposed MSAiA achieves superior deblurring performance compared with the state-of-the-art methods.

MGA-Net: A novel mask-guided attention neural network for precision neonatal brain imaging

In this study, we introduce MGA-Net, a novel mask-guided attention neural network, which extends the U-net model for precision neonatal brain imaging. MGA-Net is designed to extract the brain from other structures and reconstruct high-quality brain images. The network employs a common encoder and two decoders: one for brain mask extraction and the other for brain region reconstruction. A key feature of MGA-Net is its high-level mask-guided attention module, which leverages features from the brain mask decoder to enhance image reconstruction. To enable the same encoder and decoder to process both MRI and ultrasound (US) images, MGA-Net integrates sinusoidal positional encoding. This encoding assigns distinct positional values to MRI and US images, allowing the model to effectively learn from both modalities. Consequently, features learned from a single modality can aid in learning a modality with less available data, such as US. We extensively validated the proposed MGA-Net on diverse and independent datasets from varied clinical settings and neonatal age groups. The metrics used for assessment included the DICE similarity coefficient, recall, and accuracy for image segmentation; structural similarity for image reconstruction; and root mean squared error for total brain volume estimation from 3D ultrasound images. Our results demonstrate that MGA-Net significantly outperforms traditional methods, offering superior performance in brain extraction and segmentation while achieving high precision in image reconstruction and volumetric analysis. Thus, MGA-Net represents a robust and effective preprocessing tool for MRI and 3D ultrasound images, marking a significant advance in neuroimaging that enhances both research and clinical diagnostics in the neonatal period and beyond.

Behavioral and neural mechanisms of face-specific attention during goal-directed visual search.

Goal-directed visual attention is a fundamental cognitive process that enables animals to selectively focus on specific regions of the visual field while filtering out irrelevant information. However, given the domain specificity of social behaviors, it remains unclear whether attention to faces versus non-faces recruits different neurocognitive processes. In this study, we simultaneously recorded activity from temporal and frontal nodes of the attention network while macaques performed a goal-directed visual search task. V4 and inferotemporal (IT) visual category-selective units, selected during cue presentation, discriminated fixations on targets and distractors during the search, but were differentially engaged by face and house targets. V4 and IT category-selective units also encoded fixation transitions and search dynamics. Compared to distractors, fixations on targets reduced spike-LFP coherence within the temporal cortex. Importantly, target-induced desynchronization between the temporal and prefrontal cortices was only evident for face targets, suggesting that attention to faces differentially engaged the prefrontal cortex. We further revealed bidirectional theta influence between the temporal and prefrontal cortices using Granger causality, which was again disproportionate for faces. Finally, we showed that the search became more efficient with increasing target-induced desynchronization. Together, our results suggest domain specificity for attending to faces and an intricate interplay between visual attention and social processing neural networks.Significance Statement Visual attention stands as a cornerstone in the tapestry of visual perception. This study explores the neurocognitive mechanisms underlying goal-directed visual attention, specifically in the context of social versus non-social stimuli. By simultaneously recording neural activity from temporal and frontal nodes of the attention network in macaques, we elucidated how attentional processes differed when directed towards social or non-social targets. Our findings revealed distinct neural signatures for social versus non-social stimuli, suggesting domain specificity in the allocation of attentional resources. Moreover, we demonstrated an intricate interplay between visual attention and social processing neural networks, highlighting the complexity of social cognition in primates. These insights advance our understanding of the neural basis of social attention in primates.

ANP-Net: attention neural processes network for semi-supervised pavement defect semantic segmentation

ABSTRACT The extensive and rapid development of road transportation systems across diverse geographical landscapes underscores the imperative for efficient maintenance strategies. Pavement cracks, which expedite surface degradation, pose substantial threats to road longevity and vehicular stability, thereby highlighting the critical importance of precise detection of pavement distress images for automated road maintenance. Accurate detection not only mitigates maintenance costs but also fortifies road safety and optimizes repair operations. In contrast to traditional fully-supervised semantic segmentation algorithms, which rely heavily on extensive annotated data and neglect model prediction uncertainty, we introduce a novel Attention-Neural Process Network (ANP-Net). This attention-based semi-supervised semantic segmentation framework seamlessly integrates neural processes with attention mechanisms, presenting a robust solution for pavement defect detection. ANP-Net leverages attention-enhanced neural processes to mimic random processes through finite-dimensional marginal distributions, enabling probabilistic modeling of encoded features. It introduces a novel semi-supervised contrastive loss based on low-confidence predictions, categorizing pseudo-labeled pixels according to their entropy values and computing contrastive losses accordingly. To transcend the limitations of teacher-student networks that solely rely on loss functions and weight smoothing for interaction, ANP-Net innovatively employs a dual-path cross-attention mechanism, fostering more effective information exchange between the two networks

Construction of Cultural Heritage Knowledge Graph Based on Graph Attention Neural Network

To address the challenges posed by the vast and complex knowledge information in cultural heritage design, such as low knowledge retrieval efficiency and limited visualization, this study proposes a method for knowledge extraction and knowledge graph construction based on graph attention neural networks (GAT). Using Tang Dynasty gold and silver artifacts as samples, we establish a joint knowledge extraction model based on GAT. The model employs the BERT pretraining model to encode collected textual knowledge data, conducts sentence dependency analysis, and utilizes GAT to allocate weights among entities, thereby enhancing the identification of target entities and their relationships. Comparative experiments on public datasets demonstrate that this model significantly outperforms baseline models in extraction effectiveness. Finally, the proposed method is applied to the construction of a knowledge graph for Tang Dynasty gold and silver artifacts. Taking the Gilded Musician Pattern Silver Cup as an example, this method provides designers with a visualized and interconnected knowledge collection structure.

Brain tumor image segmentation method using hybrid attention module and improved mask RCNN

To meet the needs of automated medical analysis of brain tumor magnetic resonance imaging, this study introduces an enhanced instance segmentation method built upon mask region-based convolutional neural network. By incorporating squeeze-and-excitation networks, a channel attention mechanism, and concatenated attention neural network, a spatial attention mechanism, the model can more adeptly focus on the critical regions and finer details of brain tumors. Residual network-50 combined attention module and feature pyramid network as the backbone network to effectively capture multi-scale characteristics of brain tumors. At the same time, the region proposal network and region of interest align technology were used to ensure that the segmentation area matched the actual tumor morphology. The originality of the research lies in the deep residual network that combines attention mechanism with feature pyramid network to replace the backbone based on mask region convolutional neural network, achieving an improvement in the efficiency of brain tumor feature extraction. After a series of experiments, the precision of the model is 90.72%, which is 0.76% higher than that of the original model. Recall was 91.68%, an increase of 0.95%; Mean Intersection over Union was 94.56%, an increase of 1.39%. This method achieves precise segmentation of brain tumor magnetic resonance imaging, and doctors can easily and accurately locate the tumor area through the segmentation results, thereby quickly measuring the diameter, area, and other information of the tumor, providing doctors with more comprehensive diagnostic information.

EEG Based Emotion Analysis using Reinforced Spatio-Temporal Attentive Graph Neural and Context Net Techniques

EEG-based emotion Classification is used to distinguish and observe the mental state or emotions. Emotion classification using EEG is used for medical, security, and various other purposes. There are several deep learning and machine learning techniques that are used to classify EEG emotion signals. They do not provide sufficient accuracy and have higher complexity and a high error rate. In this manuscript, a novel Reinforced Spatio-Temporal Attentive Graph Neural Networks (RSTAGNN)and ContextNet for emotion classification with EEG signals is proposed (RSTAGNN-ContextNet-GWOA-EEG-EA). Here the input EEG signals are taken from two benchmark datasets namely as DEAP and K-EmoCon datasets. Then the input EEG signals are pre-processed and the features are extracted using ContextNet with Global principal component Analysis (GPCA). After that, the EEG signal emotions are classified using Reinforced Spatio-Temporal Attentive Graph Neural Networks (RSTAGNN) method. Then the weight parameters of the RSTAGNN are optimized using Glowworm Swarm Optimization Algorithm (GWOA). The proposed model classifies the EEG signal emotions with high accuracy. The performance of the proposed method using DEAP dataset attains higher accuracy by 24.05%, 12.64%relatedto existing systems, like DWT-SVM-EEG-EA-DEAP and GCNN-LSTM-EEG-EA-DEAP respectively and the performance of the proposed system using K-EmoCon dataset attains higher accuracy 32.64%, 15.65% related to existing systems, like SigRep-EEG-EA-K- EEG Based Emotion Analysis using Reinforced Spatio-Temporal Attentive Graph Neural and Context Net Techniques on Deap and K- EmoCon Article and CAT-EEG-EA-K-EmoCon respectively.

Facial Video-Based Non-Contact Stress Recognition Utilizing Multi-Task Learning With Peak Attention.

Negative emotional states, such as anxiety and depression, pose significant challenges in contemporary society, often stemming from the stress encountered in daily activities. Stress (state or level) recognition is a crucial prerequisite for effective stress management and intervention. Presently, wearable devices have been employed to capture physiological signals and analyze stress states. However, their constant skin contact can lead to discomfort and disturbance during prolonged monitoring. In this paper, a peak attention-based multitasking framework is presented for non-contact stress recognition. The framework extracts rPPG signals from RGB facial videos, utilizing them as inputs for a novel multi-task attentional convolutional neural network for stress recognition (MTASR). It incorporates peak detection and HR estimation as auxiliary tasks to facilitate stress recognition. By leveraging multi-task learning, MTASR can utilize information related to stress physiological responses, thereby enhancing feature extraction efficiency. For stress recognition, two binary classification tasks are applied: stress state recognition and stress level recognition. The model is validated on the UBFC-Phys public dataset and demonstrates an accuracy of 94.33% for stress state recognition and 83.83% for stress level recognition. The proposed method outperforms the dataset's baseline methods and other competing approaches.

A Frequency-Based Attention Neural Network and Subject-Adaptive Transfer Learning for sEMG Hand Gesture Classification

A Frequency-Based Attention Neural Network and Subject-Adaptive Transfer Learning for sEMG Hand Gesture Classification

Framework for content server placement using integrated learning in content delivery network

<p>Content placement is a significant concern in content delivery networks (CDN), irrespective of various evolving studies. Existing methodologies showcase various significant unaddressed issues concerning content placement approaches' complexities. Therefore, the proposed study presents a novel computational framework towards dynamic content placement strategy using a novel integrated machine learning approach. Simplified mathematical modelling is used to formulate and solve the content placement problem. At the same time, reinforcement learning and the sequential attentional neural network have been utilized to optimize the decision-making towards placement of content servers. Designed and assessed over a Python environment, the proposed scheme is witnessed to exhibit 35% reduced bandwidth utilization, 20% reduced delay, 23% reduced computational resource utilization, and 28% reduced algorithm processing time in contrast to existing predictive content placement schemes.</p>

A dual incentive mechanism based on graph attention neural network and contract in mobile opportunistic networks

In mobile opportunistic networks, messages are transmitted through opportunistic contacts between nodes. Hence, the successful delivery of messages heavily relies on the mutual cooperation among nodes in the network. However, due to limited network resources such as node energy and cache space, nodes tend to be selfish, and they are unwilling to actively participate in message forwarding. In response to this challenge, lots of incentive mechanisms have been proposed. However, most of them rely on single incentives, there are issues such as inadequate handling of selfish nodes and vulnerability to malicious attacks, which ultimately lead to poor incentive effects. Therefore, in this paper, a Dual Incentive mechanism based on Graph attention neural network and Contract (DIGC) is introduced to encourage active participation of network nodes in data transmission. This incentive mechanism is divided into two steps. In the first step, the graph attention neural network is used to evaluate the reputation of nodes to achieve the goal of reputation-based incentive, and blockchain is employed to store and manage node reputation to ensure security and transparency. In the second step, an incentive based on contract theory is introduced, where personalized contracts were designed based on the different resources owned by nodes, thereby establishing a reward mechanism to encourage collaborative transmission. Extensive simulations based on two real-life mobility traces have been done to evaluate the performance of our DIGC compared with other existing incentive mechanisms. The results show that, our proposed mechanism can greatly improve throughput and reduce average delay while ensuring the overall delivery performance of the network.

Road Network Intelligent Selection Method Based on Heterogeneous Graph Attention Neural Network

Selecting road networks in cartographic generalization has consistently posed formidable challenges, driving research toward the application of intelligent models. Despite previous efforts, the accuracy and connectivity preservation in these studies, particularly when dealing with road types of similar sample sizes, still warrant improvement. To address these shortcomings, we introduce a Heterogeneous Graph Attention Network (HAN) for road selection, where the feature masking method is initially utilized to assess the significance of road features. Concentrating on the most relevant features, two meta-paths are introduced within the HAN framework: one for aggregating features of the same road type within the first-order neighborhood, emphasizing local connectivity, and another for extending this aggregation to the second-order neighborhood, capturing a broader spatial context. For a comprehensive evaluation, we use a set of metrics considering both quantitative and qualitative aspects of the road network. On road types with similar sample sizes, the HAN model outperforms other models in both transductive and inductive tasks. Its accuracy (ACC) is higher by 1.62% and 0.67%, and its F1-score is higher by 1.43% and 0.81%, respectively. Additionally, it enhances the overall connectivity of the selected network. In summary, our HAN-based method provides an advanced solution for road network selection, surpassing previous approaches in terms of accuracy and connectivity preservation.

A hybrid transposed attention based deep learning model for wearable and explainable stress recognition

Stress is a prevalent issue in modern society which affects a large percentage of the population. Stress has negative effects on human daily lives such as reduced memory and concentration levels, forgetfulness, restlessness, and weakness. Although it is challenging to extinguish stress from human lives, monitoring and controlling its consequences can be implemented. Current methods for stress monitoring often lack interpretability of results and making it difficult to provide an interpretable system for immediate mental health counseling and intervention. Wearable devices are of growing interest due to the impact of the COVID-19 pandemic on mental health, as wearables provide monitoring of physiological parameters which helps in measuring stress levels. Here, an investigation on the feasibility of using wrist and chest-based multimodal physiological signals collected from wearables is executed for stress classification and identification of underlying modalities that impact the stress. Multimodalities from wrist and chest sensors are used to examine the three-class classification, baseline, stress and amusement. For improved and visualized effects, the multimodal signals from wearables were transformed into images. Then, a hybrid model was introduced, which comprised of attentive convolution neural network (CNN), transposed attentive CNN connections with long-short term memory (LSTM) and followed by attention layer for efficient feature extraction and improved classification accuracy. Results found that the proposed model with images provided 97 % accuracy and 96 % F1-score for chest wearable and 94 % accuracy and 93 % F1-score for wrist wearable data. Additionally, the post-hoc explainability approach (Local interpretable model agnostic applications, LIME) provided visual representations of contributing features from each signal for each class. LIME shows electrodermal activity (EDA), temperature (TEMP) and respiration (RESP) to be the significant factor in stress recognition from chest wearable and blood volume pulse (BVP) and EDA from wrist wearable. Additionally, increasing the number of features of the explainable model influences the modalities influence on the model explainability. The results establish a benchmark for explainable stress recognition using different sensor data.

SFT-SGAT: A semi-supervised fine-tuning self-supervised graph attention network for emotion recognition and consciousness detection

Emotional recognition is highly important in the field of brain-computer interfaces (BCIs). However, due to the individual variability in electroencephalogram (EEG) signals and the challenges in obtaining accurate emotional labels, traditional methods have shown poor performance in cross-subject emotion recognition. In this study, we propose a cross-subject EEG emotion recognition method based on a semi-supervised fine-tuning self-supervised graph attention network (SFT-SGAT). First, we model multi-channel EEG signals by constructing a graph structure that dynamically captures the spatiotemporal topological features of EEG signals. Second, we employ a self-supervised graph attention neural network to facilitate model training, mitigating the impact of signal noise on the model. Finally, a semi-supervised approach is used to fine-tune the model, enhancing its generalization ability in cross-subject classification. By combining supervised and unsupervised learning techniques, the SFT-SGAT maximizes the utility of limited labeled data in EEG emotion recognition tasks, thereby enhancing the model’s performance. Experiments based on leave-one-subject-out cross-validation demonstrate that SFT-SGAT achieves state-of-the-art cross-subject emotion recognition performance on the SEED and SEED-IV datasets, with accuracies of 92.04% and 82.76%, respectively. Furthermore, experiments conducted on a self-collected dataset comprising ten healthy subjects and eight patients with disorders of consciousness (DOCs) revealed that the SFT-SGAT attains high classification performance in healthy subjects (maximum accuracy of 95.84%) and was successfully applied to DOC patients, with four patients achieving emotion recognition accuracies exceeding 60%. The experiments demonstrate the effectiveness of the proposed SFT-SGAT model in cross-subject EEG emotion recognition and its potential for assessing levels of consciousness in patients with DOC.

Enhancing hyperspectral image classification with graph attention neural network

Enhancing hyperspectral image classification with graph attention neural network

Designing an ultra-wideband directional antipodal Vivaldi antenna with U-slots for biomedical applications using an optimized attention network

Abstract This research introduces a compact Ultra-Wideband (UWB) antipodal Vivaldi antenna with a U-slot tailored for biomedical applications. Utilizing an elliptical tapered patch variation on a 50*32 mm2 substrate achieves its compact design. Parameter optimization, employing the Multi-Layer Stacked Shallow Attention Neural Network (MLSSANN) with Adaptive Gannet Optimization Algorithm (AGOA), ensures accurate predictions and efficient exploration of the design space. Performance analysis includes metrics like gain, directivity, reflection coefficients, return loss and radiation efficiency. The U-slot variant exhibits higher gain peaking at 24.5 GHz (10 dB) compared to the standard variant peaking near 37 GHz (13.7 dB). The U-slot antenna also shows improved directivity and return loss, with a 13.89 % enhancement in return loss at 3.7 GHz. Moreover, the addition of the slot shifts the lower cut-off frequency from 1,580 MHz to 740 MHz, reducing the antenna size by 41 % while maintaining acceptable radiation characteristics.