Attention Neural Network Research Articles

Attentional visual graph neural network based facial expression recognition method

Attentional visual graph neural network based facial expression recognition method

INA-Net: An integrated noise-adaptive attention neural network for enhanced medical image segmentation

Skin cancer, a growing health concern, poses a significant threat. Automated segmentation of skin lesions is crucial for precise clinical diagnosis and treatment. However, challenges such as hair interference, vascular occlusion, and ambiguous lesion boundaries in dermoscopic images complicate accurate segmentation. Recent research has focused on capturing multiscale-enhanced global information across spatial and channel dimensions to improve the segmentation of ambiguous boundaries. Despite these advancements, performance remains suboptimal in the presence of complex disturbances. To address this, we propose INA-Net, an Integrated Noise-Adaptive Attention Neural Network for enhanced medical image segmentation. INA-Net proposes a lightweight noise-range attention module that integrates local and simulated noise features with global information to improve robustness against complex mixed noises. Additionally, our proposed Edge-aware Spatial Attention (ESA) and Multi-scale Channel Attention (MCA) enhance feature information and boundary perception. The proposed Dynamic Noise Encoding module and Fourier Wavelet Analysis (FW-Parser) further improve robustness by handling high-frequency noise components. INA-Net effectively manages hair interference, vascular occlusion, and ambiguous lesion boundaries, thereby providing critical support for accurate lesion identification. Evaluations on the ISIC2016, ISIC2018, and PH2 datasets demonstrate outstanding performance, with IOU scores of 87.53%, 84.74%, and 85.30%, respectively. These results surpass those of other models with comparable complexity and size.

Learning to Allocate Time-Bound and Dynamic Tasks to Multiple Robots Using Covariant Attention Neural Networks

Abstract In various applications of multi-robotics in disaster response, warehouse management, and manufacturing, tasks that are known a priori and tasks added during run time need to be assigned efficiently and without conflicts to robots in the team. This multi-robot task allocation (MRTA) process presents itself as a combinatorial optimization (CO) problem that is usually challenging to be solved in meaningful timescales using typical (mixed)integer (non)linear programming tools. Building on a growing body of work in using graph reinforcement learning to learn search heuristics for such complex CO problems, this paper presents a new graph neural network architecture called the covariant attention mechanism (CAM). CAM can not only generalize but also scale to larger problems than that encountered in training, and handle dynamic tasks. This architecture combines the concept of covariant compositional networks used here to embed the local structures in task graphs, with a context module that encodes the robots’ states. The encoded information is passed onto a decoder designed using multi-head attention mechanism. When applied to a class of MRTA problems with time deadlines, robot ferry range constraints, and multi-trip settings, CAM surpasses a state-of-the-art graph learning approach based on the attention mechanism, as well as a feasible random-walk baseline across various generalizability and scalability tests. Performance of CAM is also found to be at par with a high-performing non-learning baseline called BiG-MRTA, while noting up to a 70-fold improvement in decision-making efficiency over this baseline.

TLSTSRec: Time-aware long short-term attention neural network for sequential recommendation

TLSTSRec: Time-aware long short-term attention neural network for sequential recommendation

Torture Exposure Modulates Cognitive Control and Attention Neural Network Connectivity During a Go/NoGo Task

BackgroundTorture trauma is characterized by intentional uncontrollable acts, but the long-term effects of torture exposure on cognitive control brain mechanisms are unknown. MethodsA final sample of 33 torture survivors (TSs) and 44 non-TSs, all with a refugee background, completed a Go/NoGo response inhibition task during functional magnetic resonance imaging scanning. Data-driven independent component analysis identified active networks across the task and on Go, NoGo, and error of commission trials. Groups were compared on within-/between-network connectivity while for demographic and psychological symptom covariates were controlled for. Secondary analyses investigated whether network connectivity moderated the associations between torture exposure and severity on fear (e.g., re-experiencing) and dysphoria (e.g., anhedonia) posttraumatic stress disorder symptoms. ResultsThe TS group exhibited decreased connectivity (compared with the non-TS control group) within the posterior default mode network (specifically the left precuneus) and auditory-motor network (specifically the right superior temporal gyrus) and reduced connectivity between the dorsomedial frontal network and dorsal attention network across the Go/NoGo task. The TS group also showed more negative ventral attention network connectivity during NoGo (i.e., inhibition) trials. No behavioral effects were observed. Secondary analyses revealed that the association between torture exposure and elevated posttraumatic stress disorder dysphoria (not fear) symptoms was moderated by reduced connectivity in the right superior temporal gyrus and between the dorsomedial frontal network and the dorsal attention network. ConclusionsResponse inhibition, attention, and motor networks appear to be less connected in TSs, which may be specifically linked to posttraumatic stress disorder dysphoria symptom profiles. The findings suggest that targeting cognitive control processes may hold promise for alleviating posttraumatic symptoms among survivors of torture.

DAT-Net: Filling of missing temperature values of meteorological stations by data augmentation attention neural network

Abstract For a long time, filling in the missing temperature data from meteorological stations has been crucial for researchers in analyzing climate variation cases. In previous studies, people have attempted to solve this problem by using interpolation and deep learning methods. Through extensive case studies, it is observed that the data utilization rate of convolutional neural networks based on PConv is low at a high missing rate, which will result in the poor filling performance of each model at a high missing rate. To solve these problems, a Data Augmentation Attention Neural Network (DAT-Net) is presented. DAT Net uses encoder and decoder structures, which include a data augmentation training mechanism (DAM) to enhance model training. In addition, a time encoder (TED) has been developed to assist the model in learning the temporal dependencies of the data. To evaluate DAT-Net, 75% and 85% of experiments were performed, while comparisons were made with Linear, NLinear, DLinear, PatchTST, and GSTA-Net. The results showed that when the missing rate was 75%, DAT-Net decreased by 55.22%, 55.05%, 55.18%, 28.73%, and 12.35% on MAE and 54.08%, 53.88%, 54.08%, 35.48% and, 14.51% on RMSE, R 2 increased by 3.80%, 3,75%, 3.68%, 0.55%, and 0.27%, respectively.

Attribute Diversity Aware Community Detection on Attributed Graphs Using Three-View Graph Attention Neural Networks

Community detection is a fundamental yet important task for characterizing and understanding the structure of attributed graphs. Existing methods mainly focus on the structural tightness and attribute similarity among nodes in a community. However, grouping numerous semantically homogeneous nodes will result in information cocoons and thus reduce the robustness of community structure and the efficiency of node collaboration in real-world applications, such as recommendation systems and collaboration networks. Since nodes with closer connections tend to be more similar, finding communities with dense structures and diverse attributes poses great challenges to mining latent relationships between the graph structure and attribute distribution. To our best knowledge, very little research has been conducted to address this challenge. In this article, we propose a novel three-view graph attention neural networks (TvGANN) model to formally address the attribute diversity aware community detection problem. TvGANN reveals correlations between the graph structure and attributes distribution from the perspective of node organization, attribute co-occurrence, and the node-attribute interaction. It effectively captures structural features and attributes distribution by feeding a structural network and an attribute co-occurrence network into graph attention modules through the encoder–decoder framework. It also learns heterogeneous information by feeding a network into a meta-node attention module. Then, it fuzes the three modules and clusters the embedding representations through a Student's t -distribution approach, which iteratively refines the clustering results. The experiments show that our method not only improves the quality in dense community detection but also performs efficiently for attributed graphs.

Hybrid modeling for vehicle lateral dynamics via AGRU with a dual-attention mechanism under limited data

A precise vehicle dynamics model is critical for simulation and algorithm testing. Neural networks have been widely used to build high-fidelity vehicle dynamics models due to the excellent learning ability. However, data starvation is a common phenomenon in neural networks. With limited data, it is difficult for neural networks to achieve precise predictions. To address these problems, a hybrid model combining physics and dual attention neural networks is developed to model vehicle lateral dynamics. First, due to the interpretability of the physical model, linear lateral dynamics model is regarded as a prior model. However, due to the imperfect prior knowledge, there are residuals between the prior and the actual vehicle dynamics. Therefore, neural networks are used to characterize the residuals to achieve recalibration of vehicle dynamics model. Modeling vehicle residual dynamics with neural networks is a time series forecasting problem. The GRU with a dual attention mechanism and adaptive initial hidden states (DA-AGRU) is designed to capture spatial and temporal correlations in the vehicle dynamics data. In particular, considering the unique auto regressive structure of the hybrid model, a spatial attention mechanism with a feature fusion module is designed, so as to globally compute the weights of different channel features. The dataset used to train and validate the model is recorded from a vehicle platform, and the experimental results show that the proposed hybrid model can accurately predict vehicle dynamics states in a data-scarce environment.

Harnessing graph convolutional neural networks for identification of glassy states in metallic glasses

Graph Convolutional Neural Networks (GCNNs) have emerged as powerful tools for analyzing materials. In this study, we employ GCNNs to examine structural characteristics of CuZr metallic glasses (MGs) and identify their states. We use molecular dynamics to simulate the quenching process of CuZr, using cooling rates ranging from 109 to 1015 K/s to produce six unique glassy states. For each state, we create a dataset comprising 1,800 distinct samples. We evaluate the effectiveness of various GCNNs, including Graph Attention Neural Network (GANN), Graph Sample and AggreGatE (GraphSAGE), Graph Isomorphism Network (GIN), and Relational Graph Convolutional Neural Network (RGCN). GANN and GraphSAGE demonstrate comparable performance, achieving an overall accuracy of 81 % in classifying the MG states. These results underscore the potential of GCNNs to detect subtle structural variances in disordered materials and point to broader application of deep learning in the analysis of MGs and other amorphous substances.

Human Gait phases recognition based on multi-source data fusion and BILSTM attention neural network

A human gait recognition algorithm with deep learning based on multi-source data fusion is proposed to assist exoskeleton realizing complex human-exoskeleton cooperative motion. A lightweight gait acquisition device simultaneously acquires three different types of sensor signals, i.e., thigh surface electromyography (sEMG), ground reaction force (GRF) of human feet and two joints motion information. The sample data is obtained by many multi-scene gait experiments involving stand still, walk up/down stairs, and walk up/down slope, etc. The proposed algorithm recognizes multiple gait phases (27 classes) in different motion patterns by using short-time gait data, which has two typical advantages: (1) Based on multi-source sensor distributed in human body, this wearable device is constructed for ease of use to address several Subjects (different weights and heights) with low cost and light weight. (2) The critical features of gait data are identified by Bi-directional Long Short-Term Memory (BILSTM) attention neural network with higher recognition accuracy than other state-of-art recognition methods, especially in arbitrary gait switching durations.

Aberration Measurement from Crystalline Ronchigrams with an Attention Neural Network

Aberration Measurement from Crystalline Ronchigrams with an Attention Neural Network

Automatic defect detection for electrofusion joints of high-density polyethylene via spatial sequence to sequence attention neural network

High-density polyethylene pipelines have been widely used in various industries owing to their inherent advantages, such as resistance to harsh environments. However, a variety of defects are caused by welding process such as electrofusion technology. Therefore, detecting defects is necessary to guarantee welding quality and safety. The existing methods of detecting defects mainly neglect the internal spatial characteristics and depend on many rules. In this paper, detecting defect is regarded as a sequence labelling problem, and a novel spatial sequence to sequence attentive neural network is proposed to automatically identification defects via the natural spatial features of electrofusion joint. The proposed network utilizes the relative position between the neighbour resistance wire and possible defect to generate the vector representation, and expands the vectors through multivariate Gaussian probability distribution to highlight the vital sequence features related to defect. Self-attention is employed to capture the prominent distinguishing information between normal resistance wire and possible defect. The experimental results on the actual dataset collecting from electrofusion joints of high-density polyethylene illustrate that expanded vector is useful to focus on crucial features and self-attention promotes the identification ability of the proposed network, achieving the F1-score of 92.28 %.

Recognition of acoustic vortex fields based on a convolutional attention neural network

Recognition of acoustic vortex fields based on a convolutional attention neural network

Multi-Level Attention with 2D Table-Filling for Joint Entity-Relation Extraction

Joint entity-relation extraction is a fundamental task in the construction of large-scale knowledge graphs. This task relies not only on the semantics of the text span but also on its intricate connections, including classification and structural details that most previous models overlook. In this paper, we propose the incorporation of this information into the learning process. Specifically, we design a novel two-dimensional word-pair tagging method to define the task of entity and relation extraction. This allows type markers to focus on text tokens, gathering information for their corresponding spans. Additionally, we introduce a multi-level attention neural network to enhance its capacity to perceive structure-aware features. Our experiments show that our approach can overcome the limitations of earlier tagging methods and yield more accurate results. We evaluate our model using three different datasets: SciERC, ADE, and CoNLL04. Our model demonstrates competitive performance compared to the state-of-the-art, surpassing other approaches across the majority of evaluated metrics.

Enhancing prior-knowledge-based control with deep attention neural network for outlet moisture content of cut-tobacco

Monitoring and controlling the drying process is crucial, particularly for rotary dryers, to ensure optimal quality and efficiency. However, due to the process’s nonlinear behavior, delays, and input disturbances, achieving this can be quite challenging. Previous research has not adequately explored online forecasting for rotary dryer control in real production settings. To bridge this gap, our study focuses on tobacco drying in an actual plant and presents a novel online prediction model for moisture content. We conducted comparative experiments to validate the model’s practical application in production settings and demonstrate its effectiveness in enhancing drying control. Our approach, called MCTFormer, has proven to be scalable to high-dimensional problems. The results of our study indicate a 45% improvement in product quality through accurate prediction of outlet moisture content, better control over moisture levels, and reduced energy consumption.

Multi-attribute weighted convolutional attention neural network for multiuser physical layer authentication in IIoT

Compared with upper layer authentication, physical layer authentication (PLA) is essential in unmanned Industrial Internet of Things (IIoT) scenarios, owing to its low complexity and lightweight. However, in dynamic environments, as the amount of users expands, the accuracy of single-attribute-based authentication decreases drastically, which becomes an urgent issue for IIoT. Accordingly, this paper proposes a novel multi-attribute-based convolutional attention neural network (CANN) for multiuser PLA. Using characteristics such as amplitude, phase, and delay, the multiple attributes from a real industrial scene are first constructed into three-dimensional matrices fed into CANN. Then, attention blocks are designed to learn the correlation between attributes and extract the attribute parts that are more instrumental in the CANN to improve authentication accuracy. In addition, to avoid confusing multiple users, a center confidence loss is introduced, which adaptively adjusts the weight of the center loss and works together with the softmax loss to train the CANN. The effectiveness of the proposed CANN-based multiuser PLA and center confidence loss is supported by experimental results. Compared with the recently proposed latent perturbed convolutional neural network (LPCNN), the CANN-based scheme improves the authentication accuracy by 8.11%, which is superior to the existing learning-based approaches. As the CANN is further trained with the loss function that combines center confidence loss, the authentication accuracy can be improved by at least 2.22%.

Deep Learning-Based Dynamic Risk Prediction of Venous Thromboembolism for Patients With Ovarian Cancer in Real-World Settings From Electronic Health Records.

Patients with epithelial ovarian cancer (EOC) have an elevated risk for venous thromboembolism (VTE). To assess the risk of VTE, models were developed by statistical or machine learning algorithms. However, few models have accommodated deep learning (DL) algorithms in realistic clinical settings. We aimed to develop a predictive DL model, exploiting rich information from electronic health records (EHRs), including dynamic clinical features and the presence of competing risks. We extracted EHRs of 1,268 patients diagnosed with EOC from January 2007 through December 2017 at the National Cancer Center, Korea. DL survival networks using fully connected layers, temporal attention, and recurrent neural networks were adopted and compared with multi-perceptron-based classification models. Prediction accuracy was independently validated in the data set of 423 patients newly diagnosed with EOC from January 2018 to December 2019. Personalized risk plots displaying the individual interval risk were developed. DL-based survival networks achieved a superior area under the receiver operating characteristic curve (AUROC) between 0.95 and 0.98 while the AUROC of classification models was between 0.85 and 0.90. As clinical information benefits the prediction accuracy, the proposed dynamic survival network outperformed other survival networks for the test and validation data set with the highest time-dependent concordance index (0.974, 0.975) and lowest Brier score (0.051, 0.049) at 6 months after a cancer diagnosis. Our visualization showed that the interval risk fluctuating along with the changes in longitudinal clinical features. Adaption of dynamic patient clinical features and accounting for competing risks from EHRs into the DL algorithms demonstrated VTE risk prediction with high accuracy. Our results show that this novel dynamic survival network can provide personalized risk prediction with the potential to assist risk-based clinical intervention to prevent VTE among patients with EOC.

Behavioral and neural mechanisms of face-specific attention during goal-directed visual search.

Goal-directed visual attention is a fundamental cognitive process that enables animals to selectively focus on specific regions of the visual field while filtering out irrelevant information. However, given the domain specificity of social behaviors, it remains unclear whether attention to faces versus non-faces recruits different neurocognitive processes. In this study, we simultaneously recorded activity from temporal and frontal nodes of the attention network while macaques performed a goal-directed visual search task. V4 and inferotemporal (IT) visual category-selective units, selected during cue presentation, discriminated fixations on targets and distractors during the search, but were differentially engaged by face and house targets. V4 and IT category-selective units also encoded fixation transitions and search dynamics. Compared to distractors, fixations on targets reduced spike-LFP coherence within the temporal cortex. Importantly, target-induced desynchronization between the temporal and prefrontal cortices was only evident for face targets, suggesting that attention to faces differentially engaged the prefrontal cortex. We further revealed bidirectional theta influence between the temporal and prefrontal cortices using Granger causality, which was again disproportionate for faces. Finally, we showed that the search became more efficient with increasing target-induced desynchronization. Together, our results suggest domain specificity for attending to faces and an intricate interplay between visual attention and social processing neural networks.

Distinct visual processing networks for foveal and peripheral visual fields.

This study investigates the systematic differences between foveal and peripheral vision, two crucial components of visual processing essential for navigating our surroundings. By simultaneously recording from a large number of neurons in the visual attentional neural network, we revealed substantial variations in the proportion and functional characteristics of foveal and peripheral units across different brain regions. We uncovered differential modulation of functional connectivity by attention and visual selectivity, elucidated the intricate interplay between foveal and peripheral processing in spatial attention and saccade selection, and linked neural responses to search behavior. Overall, our study contributes to a deeper understanding of how the brain processes and integrates visual information for active visual behaviors.

A novel multi-feature fusion attention neural network for the recognition of epileptic EEG signals.

Epilepsy is a common chronic brain disorder. Detecting epilepsy by observing electroencephalography (EEG) is the main method neurologists use, but this method is time-consuming. EEG signals are non-stationary, nonlinear, and often highly noisy, so it remains challenging to recognize epileptic EEG signals more accurately and automatically. This paper proposes a novel classification system of epileptic EEG signals for single-channel EEG based on the attention network that integrates time-frequency and nonlinear dynamic features. The proposed system has three novel modules. The first module constructs the Hilbert spectrum (HS) with high time-frequency resolution into a two-channel parallel convolutional network. The time-frequency features are fully extracted by complementing the high-dimensional features of the two branches. The second module constructs a grayscale recurrence plot (GRP) that contains more nonlinear dynamic features than traditional RP, fed into the residual-connected convolution module for effective learning of nonlinear dynamic features. The third module is the feature fusion module based on a self-attention mechanism to assign optimal weights to different types of features and further enhance the information extraction capability of the system. Therefore, the system is named HG-SANet. The results of several classification tasks on the Bonn EEG database and the Bern-Barcelona EEG database show that the HG-SANet can effectively capture the contribution degree of the extracted features from different domains, significantly enhance the expression ability of the model, and improve the accuracy of the recognition of epileptic EEG signals. The HG-SANet can improve the diagnosis and treatment efficiency of epilepsy and has broad application prospects in the fields of brain disease diagnosis.