Attention Neural Network Research Articles

Multi‐agent trajectory prediction with adaptive perception‐guided transformers

AbstractThe ability to predict the trajectory of an autonomous vehicle accurately is crucial for safe and efficient navigation. However, predicting diverse and multimodal futures can be challenging. Recent approaches such as attention and graph neural networks have achieved state‐of‐the‐art performance by considering agent interactions and map contexts. This study focused on multi‐agent prediction using an agent‐centric approach with transformers. This enables parallel computation and a comprehensive understanding of the environment. Two main features are introduced: an adaptive receptive field (ARF) that captures the relevant surroundings for each agent, and perception encoding, which serves as spatial context embeddings. The ARF adapts to the agent's velocity and rotation, focusing attention ahead at high speeds or to the sides when it is slower. Perception encoding divides agents or lanes into levels and encodes the information of each level. This approach enables the efficient encoding of complex spatial relationships. The proposed method combines these advances with transformer modelling for multi‐agent trajectory prediction while ensuring real‐time prediction capabilities. The approach is evaluated on the Argoverse benchmark and better performance than the state‐of‐the‐art baseline is achieved. By addressing challenges such as multimodal outputs and robustness, the study enhances the safety and efficiency of autonomous driving systems by more accurately predicting trajectories.

Remaining Useful Life Prediction of Lithium-Ion Batteries: A Temporal and Differential Guided Dual Attention Neural Network

Remaining Useful Life Prediction of Lithium-Ion Batteries: A Temporal and Differential Guided Dual Attention Neural Network

IoT Route Planning Based on Spatiotemporal Interactive Attention Neural Network

IoT Route Planning Based on Spatiotemporal Interactive Attention Neural Network

Tomato Disease Detection Using Vision Transformer with Residual L1-Norm Attention and Deep Neural Networks

Tomato Disease Detection Using Vision Transformer with Residual L1-Norm Attention and Deep Neural Networks

MFCANN: A feature diversification framework based on local and global attention for human activity recognition

Human activity recognition (HAR) is a crucial detection technique extensively employed in various contexts demanding accurate identification of human actions. Currently, mainstream HAR approaches frequently rely on sequential data generated by wearable sensors, with a primary focus on extracting feature representations that align with various human activities. Nevertheless, complex and diverse human activities encompass various kinds of features and substantial variations in the features emphasized across different activities, making it challenging for current artificial intelligence methods to achieve accurate recognition. To address this challenge, this paper introduces a network framework called the Multi-Feature Combining Attention Neural Network (MFCANN). This framework incorporates diverse feature extraction, along with both local and global feature attention mechanisms. It is composed of stacked Multi-Feature Combining Attention Blocks (MFCAB) designed by us. In contrast to traditional convolutional methods, MFCAB parallelly stacks various convolutional components, enabling the extraction of a broader range of features from human activity data. Additionally, we propose the Intra-Module Attention Block (Intra-MAB) and the Inter-Module Attention Block (Inter-MAB), which simultaneously focus on local fine-grained features within module feature maps and global distinguishing features across module feature maps. This aims to achieve more targeted feature learning, reinforcing the network’s ability to distinguish between different human activities. Abundant experimental results demonstrate that the proposed MFCANN in this paper outperforms current mainstream deep learning algorithms in HAR tasks, achieving recognition accuracies of 0.9813, 0.9324, and 0.9930 on the UCI-HAR, USC-HAD, and RealWorld datasets respectively. We release our code at https://github.com/RangerHeart/MFCANN.

Graph Time-series Modeling in Deep Learning: A Survey

Time-series and graphs have been extensively studied for their ubiquitous existence in numerous domains. Both topics have been separately explored in the field of deep learning. For time-series modeling, recurrent neural networks or convolutional neural networks model the relations between values across timesteps, while for graph modeling, graph neural networks model the inter-relations between nodes. Recent research in deep learning requires simultaneous modeling for time-series and graphs when both representations are present. For example, both types of modeling are necessary for time-series classification, regression, and anomaly detection in graphs. This article aims to provide a comprehensive summary of these models, which we call graph time-series models. To the best of our knowledge, this is the first survey article that provides a picture of related models from the perspective of deep graph time-series modeling to address a range of time-series tasks, including regression, classification, and anomaly detection. Graph time-series models are split into two categories: (a) graph recurrent/convolutional neural networks and (b) graph attention neural networks. Under each category, we further categorize models based on their properties. Additionally, we compare representative models and discuss how distinctive model characteristics are utilized with respect to various model components and data challenges. Pointers to commonly used datasets and code are included to facilitate access for further research. In the end, we discuss potential directions for future research.

Enhanced model iteration algorithm with graph neural network for diffuse optical tomography.

Diffuse optical tomography (DOT) employs near-infrared light to reveal the optical parameters of biological tissues. Due to the strong scattering of photons in tissues and the limited surface measurements, DOT reconstruction is severely ill-posed. The Levenberg-Marquardt (LM) is a popular iteration method for DOT, however, it is computationally expensive and its reconstruction accuracy needs improvement. In this study, we propose a neural model based iteration algorithm which combines the graph neural network with Levenberg-Marquardt (GNNLM), which utilizes a graph data structure to represent the finite element mesh. In order to verify the performance of the graph neural network, two GNN variants, namely graph convolutional neural network (GCN) and graph attention neural network (GAT) were employed in the experiments. The results showed that GCNLM performs best in the simulation experiments within the training data distribution. However, GATLM exhibits superior performance in the simulation experiments outside the training data distribution and real experiments with breast-like phantoms. It demonstrated that the GATLM trained with simulation data can generalize well to situations outside the training data distribution without transfer training. This offers the possibility to provide more accurate absorption coefficient distributions in clinical practice.

MoSCminer: a cell subtype classification framework based on the attention neural network integrating the single-cell multi-omics dataset on the cloud.

Single-cell omics sequencing has rapidly advanced, enabling the quantification of diverse omics profiles at a single-cell resolution. To facilitate comprehensive biological insights, such as cellular differentiation trajectories, precise annotation of cell subtypes is essential. Conventional methods involve clustering cells and manually assigning subtypes based on canonical markers, a labor-intensive and expert-dependent process. Hence, an automated computational prediction framework is crucial. While several classification frameworks for predicting cell subtypes from single-cell RNA sequencing datasets exist, these methods solely rely on single-omics data, offering insights at a single molecular level. They often miss inter-omic correlations and a holistic understanding of cellular processes. To address this, the integration of multi-omics datasets from individual cells is essential for accurate subtype annotation. This article introduces moSCminer, a novel framework for classifying cell subtypes that harnesses the power of single-cell multi-omics sequencing datasets through an attention-based neural network operating at the omics level. By integrating three distinct omics datasets-gene expression, DNA methylation, and DNA accessibility-while accounting for their biological relationships, moSCminer excels at learning the relative significance of each omics feature. It then transforms this knowledge into a novel representation for cell subtype classification. Comparative evaluations against standard machine learning-based classifiers demonstrate moSCminer's superior performance, consistently achieving the highest average performance on real datasets. The efficacy of multi-omics integration is further corroborated through an in-depth analysis of the omics-level attention module, which identifies potential markers for cell subtype annotation. To enhance accessibility and scalability, moSCminer is accessible as a user-friendly web-based platform seamlessly connected to a cloud system, publicly accessible at http://203.252.206.118:5568. Notably, this study marks the pioneering integration of three single-cell multi-omics datasets for cell subtype identification.

Graph attention neural networks for mapping materials and molecules beyond short-range interatomic correlations

Bringing advances in machine learning to chemical science is leading to a revolutionary change in the way of accelerating materials discovery and atomic-scale simulations. Currently, most successful machine learning schemes can be largely traced to the use of localized atomic environments in the structural representation of materials and molecules. However, this may undermine the reliability of machine learning models for mapping complex systems and describing long-range physical effects because of the lack of non-local correlations between atoms. To overcome such limitations, here we report a graph attention neural network as a unified framework to map materials and molecules into a generalizable and interpretable representation that combines local and non-local information of atomic environments from multiple scales. As an exemplary study, our model is applied to predict the electronic structure properties of metal-organic frameworks (MOFs) which have notable diversity in compositions and structures. The results show that our model achieves the state-of-the-art performance. The clustering analysis further demonstrates that our model enables high-level identification of MOFs with spatial and chemical resolution, which would facilitate the rational design of promising reticular materials. Furthermore, the application of our model in predicting the heat capacity of complex nanoporous materials, a critical property in a carbon capture process, showcases its versatility and accuracy in handling diverse physical properties beyond electronic structures.

Hierarchical-taxonomy-aware and attentional convolutional neural networks for acoustic identification of bird species: A phylogenetic perspective

The study of bird populations is crucial for biodiversity research and conservation. Deep artificial neural networks have revolutionized bird acoustic recognition; however, most methods overlook inherent relationships among bird populations, resulting in the loss of biological information. To address this limitation, we propose the Phylogenetic Perspective Neural Network (PPNN), which incorporates hierarchical multilevel labels for each bird. PPNN uses a hierarchical semantic embedding framework to capture feature information at different levels. Attention mechanisms are employed to extract and select common and distinguishing features, thereby improving classification accuracy. We also propose a path correction strategy to rectify inconsistent predictions. Experimental results on bird acoustic datasets demonstrate that PPNN outperforms current methods, achieving classification accuracies of 90.450%, 91.883%, and 89.950% on the Lishui-Zhejiang Birdsdata (100 species), BirdCLEF2018-Small (150 species), and BirdCLEF2018-Large (500 species) datasets respectively, with the lowest hierarchical distance of a mistake across all datasets. Our proposed method is applicable to any bird acoustic dataset and presents significant advantages as the number of categories increases.

Open‐domain event schema induction via weighted attentive hypergraph neural network

SummaryEvent schema refers to the use of a template to depict similar events, and it is a necessary prerequisite for event causality extractions. The induction of event schemas is a difficult task, especially for texts in the open domain, due to the complex and diverse manifestations of events. Previous models considered participants in event mentions are independent or compositional, ignoring the high‐order correlations among participants, which limit their capability of induce event schema. To remedy this, we propose constructing an Event Structure Hypergraph (ESH) to better utilizes the event structural information for event schema induction. In particular, we first extract event mentions from the open‐domain corpus. and then construct an ESH by representing event mentions as a hyperedges. ESH contains high‐order information between participants in event mention. To, learn event mentions representation based on ESH, we propose a weighted attentive hypergraph neural network (WHGNN) to model event high‐order correlations and then integrate node‐category weight matrix into the training of network by improving event representation. By applying jointly cluster algorithm on the event mentions representation, we can induce reliable event schemas. Experimental results on three datasets demonstrate that our approach can induce salient and high‐quality event schemas on open‐domain corpus.

Hierarchical-attention-based neural network for gait emotion recognition

Human gait is an emerging biometric feature and contains important information for long-distance emotion recognition. However, sadness and neutral emotions are easily misjudged during the recognition process, because the body posture of the two emotions is quite similar. Existing methods have difficulty in distinguishing these two emotions satisfactorily since they treat all action features equally without differentiating their contribution to recognition, or they ignore the motion information in gait that can express emotion. In this paper, we propose a novel hierarchical attention neural network, which can automatically learn the affective features contained in human motion and action, and effectively distinguish between sad and neutral emotions. The network consists of three modules: motion sentiment module (MSM), action sentiment module (ASM) and emotion classifier. Specifically, MSM is composed of a position encoder and a velocity encoder. It extracts affective features from motion information and helps to distinguish the sad and neutral emotions based on gait velocity. ASM consists of an action encoder that compresses the discriminative human actions into a latent space. Emotion classifier recognizes the emotion based on the outputs from MSM and ASM. Moreover, we present a feature preprocessing method to deal with the problem of imbalanced data categories. Experiments demonstrate that our approach enhances the discriminability of sad and neutral emotions and performs better than many state-of-the-art methods. In addition, ablation experiments further verify that both velocity and action features are important for the gait-based emotion recognition task.

Trajectory-User Linking via Hierarchical Spatio-Temporal Attention Networks

Trajectory-User Linking (TUL) is crucial for human mobility modeling by linking different trajectories to users with the exploration of complex mobility patterns. Existing works mainly rely on the recurrent neural framework to encode the temporal dependencies in trajectories, have fall short in capturing spatial-temporal global context for TUL prediction. To fill this gap, this work presents a new hierarchical spatio-temporal attention neural network, called AttnTUL , to jointly encode the local trajectory transitional patterns and global spatial dependencies for TUL. Specifically, our first model component is built over the graph neural architecture to preserve the local and global context and enhance the representation paradigm of geographical regions and user trajectories. Additionally, a hierarchically structured attention network is designed to simultaneously encode the intra-trajectory and inter-trajectory dependencies, with the integration of the temporal attention mechanism and global elastic attentional encoder. Extensive experiments demonstrate the superiority of our AttnTUL method as compared to state-of-the-art baselines on various trajectory datasets. The source code of our model is available at https://github.com/Onedean/AttnTUL .

A Novel Piecewise Cubic Hermite Interpolating Polynomial-Enhanced Convolutional Gated Recurrent Method under Multiple Sensor Feature Fusion for Tool Wear Prediction.

The monitoring of the lifetime of cutting tools often faces problems such as life data loss, drift, and distortion. The prediction of the lifetime in this situation is greatly compromised with respect to the accuracy. The recent rise of deep learning, such as Gated Recurrent Unit Units (GRUs), Hidden Markov Models (HMMs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Attention networks, and Transformers, has dramatically improved the data problems in tool lifetime prediction, substantially enhancing the accuracy of tool wear prediction. In this paper, we introduce a novel approach known as PCHIP-Enhanced ConvGRU (PECG), which leverages multiple-feature fusion for tool wear prediction. When compared to traditional models such as CNNs, the CNN Block, and GRUs, our method consistently outperformed them across all key performance metrics, with a primary focus on the accuracy. PECG addresses the challenge of missing tool wear measurement data in relation to sensor data. By employing PCHIP interpolation to fill in the gaps in the wear values, we have developed a model that combines the strengths of both CNNs and GRUs with data augmentation. The experimental results demonstrate that our proposed method achieved an exceptional relative accuracy of 0.8522, while also exhibiting a Pearson's Correlation Coefficient (PCC) exceeding 0.95. This innovative approach not only predicts tool wear with remarkable precision, but also offers enhanced stability.

Remote sensing image feature matching via graph classification with local motion consistency

ABSTRACT Feature matching is a classic challenge in the computer vision field. In this paper, we propose an innovative graph classification method based on neighborhood motion consistency to eliminate erroneous matches. Specifically, we transform the coordinates of feature matching points into vectors on a unified scale. For a given match, we construct a graph centered around the match and incorporating neighboring matches. Node attributes are designed to represent the similarity between the vector of the central node and those of its neighbors. To facilitate this, we develop a lightweight graph attention neural network dedicated to graph property classification, thereby predicting the accuracy of the match under consideration. To effectively train the model, we employ a random cropping strategy to generate a plethora of diverse graphs for classifier training. We evaluate our method on datasets encompassing translational remote sensing data, rotational and scaled remote sensing imagery produced via random cropping, and nonrigid fisheye datasets. Our algorithm demonstrates superior performance to current state-of-the-art methods.

Two-Stage Point Cloud Registration Framework Based on Graph Neural Network and Attention

In recent years, due to the wide application of 3D vision in the fields of autonomous driving, robot navigation, and the protection of cultural heritage, 3D point cloud registration has received much attention. However, most current methods are time-consuming and are very sensitive to noises and outliers, resulting in low registration accuracy. Therefore, we propose a two-stage framework based on graph neural network and attention—TSGANet, which is effective in registering low-overlapping point cloud pairs and is robust to variable noises as well as outliers. Our method decomposes rigid transformation estimation into two stages: global estimation and fine-tuning. At the global estimation stage, multilayer perceptrons are employed to estimate a seven-dimensional vector representing rigid transformation directly from the fusion of two initial point cloud features. For the fine-tuning stage, we extract contextual information through an attentional graph neural network consisting of attention and feature-enhancing modules. A mismatch-suppression mechanism is also proposed and applied to keep our method robust to partially visible data with noises and outliers. Experiments show that our method yields a state-of-the-art performance on the ModelNet40 dataset.

Pose-Graph Attentional Graph Neural Network for Lidar Place Recognition

Pose-Graph Attentional Graph Neural Network for Lidar Place Recognition

LD-Recognition: Classroom Action Recognition Based on Passive RFID

Classroom learning is one of the main ways for students to acquire knowledge because the class usually has a large number of students and it is difficult for teachers to pay attention to the learning status of each student at the same time. Therefore, mastering the learning status of each student and dealing with it is the key factor to determine the quality of course teaching, and classroom actions can accurately reflect the learning state of students. Based on this, in this article, a passive radio frequency identification (RFID)-based classroom action recognition system LD-recognition is proposed. The system pastes the label on the right side of the desk, and the learning state of the students was judged by recognizing the four movements of raising the left hand, raising the right hand, nodding off, and holding the book. The system uses a multichannel attentional graph convolutional neural network (ATGCN) to deeply learn the phase and signal strength of actions and conduct action recognition. LD-recognition verifies the accuracy of actions from different distances, different experimenters, and different network models. The experimental results show that the recognition accuracy of LD-recognition system is high, reaching 96.9% on average.

CSANet: a lightweight channel and spatial attention neural network for grading diabetic retinopathy with optical coherence tomography angiography.

Diabetic retinopathy (DR) is one of the most common eye diseases. Convolutional neural networks (CNNs) have proven to be a powerful tool for learning DR features; however, accurate DR grading remains challenging due to the small lesions in optical coherence tomography angiography (OCTA) images and the small number of samples. In this article, we developed a novel deep-learning framework to achieve the fine-grained classification of DR; that is, the lightweight channel and spatial attention network (CSANet). Our CSANet comprises two modules: the baseline model, and the hybrid attention module (HAM) based on spatial attention and channel attention. The spatial attention module is used to mine small lesions and obtain a set of spatial position weights to address the problem of small lesions being ignored during the convolution process. The channel attention module uses a set of channel weights to focus on useful features and suppress irrelevant features. The extensive experimental results for the OCTA-DR and diabetic retinopathy analysis challenge (DRAC) 2022 data sets showed that the CSANet achieved state-of-the-art DR grading results, showing the effectiveness of the proposed model. The CSANet had an accuracy rate of 97.41% for the OCTA-DR data set and 85.71% for the DRAC 2022 data set. Extensive experiments using the OCTA-DR and DRAC 2022 data sets showed that the proposed model effectively mitigated the problems of mutual confusion between DRs of different severity and small lesions being neglected in the convolution process, and thus improved the accuracy of DR classification.

Detecting Depression With Heterogeneous Graph Neural Network in Clinical Interview Transcript

Depression has an intense impact on individuals, yet many cases go undiagnosed. Thus, it is imperative to design an effective model for the automated diagnosis of depression. However, existing methods do not adequately capture contextual information in a clinical interview. Inspired by the depression diagnosis process, we propose a new perspective on detecting depression as a dialog information extraction task. Specifically, this article constructs a heterogeneous graph that models the participant’s depression state and uses the graph attention network to aggregate the pieces of depressive clues. In addition, we use the focal loss as a loss function for dealing with class imbalance by reshaping the standard cross-entropy loss. Experimental results demonstrate that our proposed model depression state extraction with heterogeneous graph attention neural network (DSE-HGAT) surpasses the baseline models on the Distress Analysis Interview Corpus-Wizard of Oz (DAIC-WOZ) dataset. Meanwhile, agreement analysis between our proposed model and the gold standard shows that it is moderate ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$</tex-math> </inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$=$</tex-math> </inline-formula> 0.528, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$</tex-math> </inline-formula> 0.05). Overall, our model is very effective in identifying depression in the clinical interview transcript, which has the potential to assist doctors with medical conditions.