Multi-head Attention Mechanism Research Articles

Dynamic inferences of crash risks in freeway merging zones: a spatio-temporal deep learning model

Freeway merging zones are critical for freeway operations and management due to potential crashes arising from complex vehicle merging behaviours. This paper investigates the application of a spatio-temporal deep learning model to infer crash risks in the zones. We first introduce a crash risk index based on Time-To-Collision and vehicle merging patterns. An innovation is that the developed spatio-temporal transformer model can analyze the evolving risk index. This model effectively captures dynamic risk features through a multi-head attention mechanism within its spatio-temporal learning components. Numerical experiments on nine inference tasks with varying spatial resolutions show improved performance of the model with lower resolution. Moreover, the ST-Transformer model is benchmarked against three advanced deep learning models, which consistently demonstrates its superiority in capturing spatio-temporal dependence in risk sequences. This investigation significantly contributes to a richer understanding of proactive traffic safety, providing valuable insights for advanced freeway management and driver assistance systems.

Integrated Extraction of Entities and Relations via Attentive Graph Convolutional Networks

For information security, entity and relation extraction can be applied in sensitive information protection, data leakage detection, and other aspects. The current approaches to entity relation extraction not only ignore the relevance and dependency between name entity recognition and relation extraction but also may result in the cumulative propagation of errors. To solve this problem, it is proposed that an end-to-end joint entity and relation extraction model based on the Attention mechanism and Graph Convolutional Network (GCN) to simultaneously extract named entities and their relationships. The model includes three parts: the detection of entity span, the construction of an entity relation weighted graph, and the inference of entity relation type. Firstly, the detection of entity spans is viewed as a sequence labeling problem, and a multi-feature fusion approach for word embedding representation is designed to calculate all entity spans in a sentence to form an entity span matrix. Secondly, the entity span matrix is employed in the Multi-Head Attention mechanism for constructing the weighted adjacency matrix of the entity relation graph. Finally, for the inference of entity relation type, considering the interaction between entities and relations, the entity span matrix and relation connection matrix are simultaneously fed into the GCN for integrated extraction of entities and relations. Our model is evaluated on the public NYT dataset, attaining a precision of 66.4%, a recall of 63.1%, and an F1 score of 64.7% for joint entity and relation extraction, significantly outperforming other approaches. Experiments demonstrate that the proposed model is helpful for inferring entities and relations, considering the interaction between entities and relations through the Attention mechanism and GCN.

Convolutional-free Malware Image Classification using Self-attention Mechanisms

Malware analysis is a critical aspect of cybersecurity, aiming to identify and differentiate malicious software from benign programmes to protect computer systems from security threats. Despite advancements in cybersecurity measures, malware continues to pose significant risks in cyberspace, necessitating accurate and rapid analysis methods. This paper introduces an innovative approach to malware classification using image analysis, involving three key phases: converting operation codes into RGB image data, employing a Generative Adversarial Network (GAN) for synthetic oversampling, and utilising a simplified Vision Transformer (ViT)-based classifier for image analysis. The method enhances feature richness and explainability through visual imagery data and addresses imbalanced classification using GAN-based oversampling techniques. The proposed framework combines the strengths of convolutional autoencoders, hybrid classifiers, and adapted ViT models to achieve a balance between accuracy and computational efficiency. As shown in the experiments, our convolutional-free approach possesses excellent accuracy and precision compared with convolutional models and outperforms CNN models on two datasets, thanks to the multi-head attention mechanism. On the Big2015 dataset, our model outperforms other CNN models with an accuracy of 0.8369 and an AUC of 0.9791. Specifically, our model reaches an accuracy of 0.9697 and an F1 score of 0.9702 on MALIMG, which is extraordinary.

Fault diagnosis of rolling bearings under variable operating conditions based on improved graph neural networks

Abstract To address the issues of low diagnostic accuracy, insufficient generalization, and poor robustness in traditional fault diagnosis methods across different equipment and varying operating conditions, this paper proposes an improved graph neural network-based fault diagnosis method for rolling bearings to enhance model performance under complex conditions. First, the optimized wavelet transform coefficient features are used as nodes, and by exploring the correlations between features, node adjacency relationships are constructed. The associations between fault modes and feature node graphs under different conditions are studied, and a fault feature graph sample set based on subgraph structures is built, providing data for the subsequent graph neural network learning. Then, a multi-head attention mechanism (MHGAT) and multi-scale feature adaptive perception pooling (MSF-ASAP) are integrated to construct a multi-head graph attention mechanism model based on multi-scale feature adaptive perception pooling (MSM-GAT). MHGAT enhances the model's ability to perceive global information by learning different features from multiple perspectives and dimensions, thus improving the model's generalization. MSF-ASAP adaptively selects and aggregates information at multiple scales, enabling the model to effectively extract key features under various operating conditions, resist noise interference, and adapt to local information changes, thus improving the model's robustness under varying conditions and noisy environments. Experimental results under multiple and continuously varying conditions demonstrate that the proposed method outperforms traditional methods in terms of diagnostic accuracy and robustness. Notably, it exhibits excellent generalization when identifying unknown conditions, achieving over 95% accuracy in recognizing new conditions and maintaining over 92.5% accuracy in noisy environments.

Intelligent Helicopter Turbine Engine Fault Diagnosis Using Multi-Head Attention

A turbine engine provides power to the helicopter, enabling the helicopter to travel and hover in the air. Since the rotorcraft operates at high altitudes, ensuring safety and maintaining a healthy operational status are crucial at all times. Therefore, a prognostics and health management (PHM) system for the turbine engine must be implemented to predict any anomalies or faults to prevent catastrophic accidents. This research proposes a novel fault diagnosis method for helicopter turbine engines based on operational data acquired from actual aircraft. First, the proposed method predicts engine torque using other operational data while accounting for uncertainty. A Bayesian regression approach is employed to predict the engine torque. The torque margin, defined as the difference between the actual torque and the estimated torque, is then used to diagnose engine faults. Specifically, a multi-head attention mechanism is incorporated to capture interactions between various engine parameters. Additionally, domain adaptation techniques are applied to enhance the model's generalization performance, ensuring robustness across diverse operating conditions. The proposed method is validated using seven different datasets, each acquired from a helicopter engine. Four datasets were used for training, while the remaining three were allocated for testing and validation. The results indicated that the proposed method accurately predicted torque. Furthermore, the fault diagnosis showed promising results, leading to a 3rd-place finish in the 2024 PHM Society Data Challenge in terms of validation score.

An Integrated TCN-CrossMHA Model for Predicting circRNA-RBP Binding Sites.

Circular RNA (circRNA) has the capacity to bind with RNA binding protein (RBP), thereby exerting a substantial impact on diseases. Predicting binding sites aids in comprehending the interaction mechanism, thereby offering insights for disease treatment strategies. Here, we propose a novel approach based on temporal convolutional network (TCN) and cross multi-head attention mechanism to predict circRNA-RBP binding sites (circTCA). First, we employ two distinct encoding methodologies to obtain two raw matrices of circRNA sequences. Then, two parallel TCN blocks extract shallow and abstract features of the two matrices separately. The fusion of the two is achieved through cross multi-head attention mechanism and after this, global expectation pooling assigns weights to the concatenated feature. Finally, the task of classifying the input sequence is entrusted to a fully connected (FC) layer. We compare circTCA with other five methods and conduct ablation experiments to demonstrate its effectiveness. We also conduct feature visualization and assess the motifs extracted by circTCA with existing motifs. All in all, circTCA is effective for binding sites prediction of circRNA and RBP.

Semantic-guided attention and adaptive gating for document-level relation extraction

In natural language processing, document-level relation extraction is a complex task that aims to predict the relationships among entities by capturing contextual interactions from an unstructured document. Existing graph- and transformer-based models capture long-range relational facts across sentences. However, they still cannot fully exploit the semantic information from multiple interactive sentences, resulting in the exclusion of influential sentences for related entities. To address this problem, a novel Semantic-guided Attention and Adaptively Gated (SAAG) model is developed for document-level relation extraction. First, a semantic-guided attention module is designed to guide sentence representation by assigning different attention scores to different words. The multihead attention mechanism is then used to capture the attention of different subspaces further to generate a document context representation. Finally, the SAAG model exploits the semantic information by leveraging a gating mechanism that can dynamically distinguish between local and global contexts. The experimental results demonstrate that the SAAG model outperforms previous models on two public datasets.

Riding Feeling Recognition Based on Multi-Head Self-Attention LSTM for Driverless Automobile

With the emergence of driverless technology, passenger ride comfort has become an issue of concern. In recent years, driving fatigue detection and braking sensation evaluation based on EEG signals have received more attention, and analyzing ride comfort using EEG signals is also a more intuitive method. However, it is still a challenge to find an effective method or model to evaluate passenger comfort. In this paper, we propose a long- and short-term memory network model based on a multiple self-attention mechanism for passenger comfort detection. By applying the multiple attention mechanism to the feature extraction process, more efficient classification results are obtained. The results show that the long- and short-term memory network using the multi-head self-attention mechanism is efficient in decision making along with higher classification accuracy. In conclusion, the classifier based on the multi-head attention mechanism proposed in this paper has excellent performance in EEG classification of different emotional states, and has a broad development prospect in brain-computer interaction.

Multi-branch fusion graph neural network based on multi-head attention for childhood seizure detection.

The most common manifestation of neurological disorders in children is the occurrence of epileptic seizures. In this study, we propose a multi-branch graph convolutional network (MGCNA) framework with a multi-head attention mechanism for detecting seizures in children. The MGCNA framework extracts effective and reliable features from high-dimensional data, particularly by exploring the relationships between EEG features and electrodes and considering the spatial and temporal dependencies in epileptic brains. This method incorporates three graph learning approaches to systematically assess the connectivity and synchronization of multi-channel EEG signals. The multi-branch graph convolutional network is employed to dynamically learn temporal correlations and spatial topological structures. Utilizing the multi-head attention mechanism to process multi-branch graph features further enhances the capability to handle local features. Experimental results demonstrate that the MGCNA exhibits superior performance on patient-specific and patient-independent experiments. Our end-to-end model for automatic detection of epileptic seizures could be employed to assist in clinical decision-making.

AGCNAF: predicting disease-gene associations using GCN and multi-head attention to fuse the similarity features

Abstract Association prediction between diseases and genes is a critical step in revealing the molecular mechanisms of diseases and developing drug treatment strategies. With the explosive growth of data in the biomedical field, how to effectively utilize these data for accurate prediction has become a hotspot and challenge in current research. To overcome the limitations of current prediction methods in dealing with complex biological network structures and feature extraction, this study proposes AGCNAF, a method that combines an unsupervised Graph Convolutional Network (GCN) and a multi-head attention mechanism. The metagraph-guided random walk strategy enables AGCNAF to capture local and high-order topological structures in the graph, while GCN is responsible for realizing deep feature extraction of these structures. By incorporating similarity features through the multi-attention mechanism, AGCNAF achieves effective integration of global and local features, which significantly improves the prediction performance. By incorporating similarity features through the multi-attention mechanism, AGCNAF achieves effective integration of global and local features, which significantly improves the prediction performance. By utilizing the machine learning binary classification model for prediction, the experimental results through five-fold cross-validation show that AGCNAF demonstrates significant advantages in prediction performance compared to existing methods, with its AUC and AUPR reaching 0.9686 and 0.9709, respectively, and the AUC up to 0.9812 under specific conditions. 
To verify the practical application value of AGCNAF, this study also conduct case studies on Alzheimer's disease, lung cancer, and breast cancer. The results further confirm the excellent performance of AGCNAF in identifying potential disease-gene associations, which opens up new possibilities for future disease-gene research.

Deep Convolutional Transformer Network for Stock Movement Prediction

The prediction and modeling of stock price movements have been shown to possess considerable economic significance within the finance sector. Recently, a range of artificial intelligence methodologies, encompassing both traditional machine learning and deep learning approaches, have been introduced for the purpose of forecasting stock price fluctuations, yielding numerous successful outcomes. Nonetheless, the identification of effective features for predicting stock movements is considered a complex challenge, primarily due to the non-linear characteristics, volatility, and inherent noise present in financial data. This study introduces an innovative Deep Convolutional Transformer (DCT) model that amalgamates convolutional neural networks, Transformers, and a multi-head attention mechanism. It features an inception convolutional token embedding architecture alongside separable fully connected layers. Experiments conducted on the NASDAQ, Hang Seng Index (HSI), and Shanghai Stock Exchange Composite (SSEC) employ Mean Absolute Error (MAE), Mean Square Error (MSE), Mean Absolute Percentage Error (MAPE), accuracy, and Matthews Correlation Coefficient (MCC) as evaluation metrics. The findings reveal that the DCT model achieves the highest accuracy of 58.85% on the NASDAQ dataset with a sliding window width of 30 days. In terms of error metrics, it surpasses other models, demonstrating the lowest average prediction error across all datasets for MAE, MSE, and MAPE. Furthermore, the DCT model attains the highest MCC values across all three datasets. These results suggest a promising capability for classifying stock price trends and affirming the DCT model’s superiority in predicting closing prices.

Short-term freeway traffic speed multistep prediction using an iTransformer model

Accurate prediction of the freeway traffic speed is important for enhancing intelligent transportation management and assisting with route planning. Current traffic speed prediction studies usually neglect predictions extending to an hour or longer. Thus, to address this gap, the inverted Transformer (iTransformer) model is utilized to predict freeway speeds across intervals from 5 to 150 minutes. The iTransformer model provides a global view by encapsulating the entire time series from each detector into variate tokens, allowing it to capture the complex patterns and dependencies that change over time. Additionally, a multi-head attention mechanism is employed to identify short-term and long-term speed patterns. This study validates the prediction performance of iTransformer by using the traffic speed data from an interstate freeway in Minnesota, comparing it against traditional traffic prediction methods and two other Transformer models. Results indicate that the iTransformer model outperforms these benchmark approaches, particularly when predicting speed over 60 minutes for peak hour periods.

A Novel WTG Method for Predicting Ship Trajectories in the Fujian Inshore Area Based on AIS Data

The increasing congestion in major global maritime routes poses significant threats to international maritime safety, exacerbated by the proliferation of large, high-speed vessels. To improve the detection of abnormal ship behavior, this research employed automatic identification system (AIS) data for ship trajectory forecasting. Traditional methods primarily focus on spatial and temporal correlations but often lack accuracy and reliability. In this study, ship path predictions were enhanced using the WTG model, which combines wavelet transform, temporal convolutional networks (TCN), and gated recurrent units (GRU). Initially, wavelet decomposition was applied to deconstruct the input trajectory time series. The TCN and GRU modules then extracted features from both the time series and the decomposed data. The predicted elements were reassembled using a multi-head attention mechanism and a pooling layer to produce the final predictions. Comparative experiments demonstrated that the WTG model surpasses other models in the accuracy of ship trajectory prediction. The model proposed in this study proves to be reliable for forecasting ship paths, which is crucial for marine traffic management and ensuring safe navigation.

Deep learning-based automated identification on vortex-induced vibration of long suspenders for the suspension bridge

Large amplitude vortex-induced vibration (VIV) of long suspenders, caused by vortex-shedding and wake flow, will decrease the service life of high-strength wires and its anchorage systems, hence listed as an essential monitoring indicator of the structural health monitoring system in the suspension bridges. Traditionally, a static vibration amplitude limitation is usually predefined and used to identify VIV from continuously monitored acceleration data of bridge suspenders. Due to the complexity of the starting vibration of the suspenders, it may not be able to flexibly adapt to the suspenders VIV under different conditions. In addition, transient but significant vibration events may not be captured since it relies only on static vibration amplitude limits without considering the temporal information of vibration. To address the above-mentioned issues, a novel framework to autonomously identify the suspenders VIV is proposed using the multimodal fusion techniques with deep neural networks. The main contributions of the methodology are: i) by calculating the wind field and vibration characteristic, two vibration response parameters are identified as indexes for the identification of suspenders VIV based on the significant difference method of big data analysis; ii) Two different modal information of time history images and power spectral density (PSD) sequences are used as inputs to the convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) to extract the suspenders vibration response features from the time and frequency domain perspectives, respectively; and iii) The multimodal information is concentrated and enhanced by the multi-head attention (MHA) mechanism to achieve automated identification of the suspenders VIV. The proposed framework is validated with data collected from a full-scale long-span suspension bridge in comparison with the base model. The results show that the proposed framework can efficiently identify the full-cycle development process of the suspenders VIV. This study provides a convenient strategy for suspenders VIV identification, which can be used for bridge management and vibration mitigation device design.

Damage identification method for jacket platform based on dual-channel model

To address the challenges posed by noisy vibration signals and underutilized time-series data in jacket platforms, this paper proposes a dual-channel damage detection method that integrates Temporal Convolutional Networks (TCN) and Gated Recurrent Units (GRU) in parallel. A multi-head attention mechanism (MA) is employed to reassign feature weights, improving detection accuracy. The optimized features are fused using the Concatenate function for the final output. Two experimental scenarios—isolated noise and ocean noise—were designed to evaluate the method. The results demonstrate that the combination of TCN, GRU and MA effectively detects damage in offshore platforms, surpassing other deep learning models. Although the method shows strong potential for real-world applications, further testing in more complex ocean environments is required to address potential limitations in handling highly variable noise patterns.

Fine-art recognition using convolutional transformers

Digital image processing is a constantly evolving field encompassing a wide range of techniques and applications. Researchers worldwide are continually developing various algorithms across multiple fields to achieve accurate image classification. Advanced computer vision algorithms are crucial for architectural and artistic analysis. The digitalization of art has significantly enhanced the accessibility and conservation of fine-art paintings, yet the risk of art theft remains a significant challenge. Improving art security necessitates the precise identification of fine-art paintings. Although current recognition systems have shown potential, there is significant scope for enhancing their efficiency. We developed an improved recognition system for categorizing fine-art paintings using convolutional transformers, specified by an attention mechanism to enhance focused learning on the data. As part of the most advanced architectures in the deep learning family, transformers are empowered by a multi-head attention mechanism, thus improving learning efficiency. To assess the performance of our model, we compared it with those developed using four pre-trained networks: ResNet50, VGG16, AlexNet, and ViT. Each pre-trained network was integrated into a corresponding state-of-the-art model as the first processing blocks. These four state-of-the-art models were constructed under the transfer learning strategy, one of the most commonly used approaches in this field. The experimental results showed that our proposed system outperformed the other models. Our study also highlighted the effectiveness of using convolutional transformers for learning image features.

Audiovisual emotion recognition based on bi-layer LSTM and multi-head attention mechanism on RAVDESS dataset

Audiovisual emotion recognition based on bi-layer LSTM and multi-head attention mechanism on RAVDESS dataset

GASCOM: Graph-based Attentive Semantic Context Modeling for Online Conversation Understanding

Online conversation understanding is an important yet challenging NLP problem which has many useful applications (e.g., hate speech detection). However, online conversations typically unfold over a series of posts and replies to those posts, forming a tree structure within which individual posts may refer to semantic context from elsewhere in the tree. Such semantic cross-referencing makes it difficult to understand a single post by itself; yet considering the entire conversation tree is not only difficult to scale but can also be misleading as a single conversation may have several distinct threads or points, not all of which are relevant to the post being considered. In this paper, we propose a Graph-based Attentive Semantic COntext Modeling (GASCOM) framework for online conversation understanding. Specifically, we design two novel algorithms that utilize both the graph structure of the online conversation as well as the semantic information from individual posts for retrieving relevant context nodes from the whole conversation. We further design a token-level multi-head graph attention mechanism to pay different attentions to different tokens from different selected context utterances for fine-grained conversation context modelling. Using this semantic conversational context, we re-examine two well-studied problems: polarity prediction and hate speech detection. Our proposed framework significantly outperforms state-of-the-art methods on both tasks, improving macro-F1 scores by 4.5% for polarity prediction and by 5% for hate speech detection. The GASCOM context weights also enhance interpretability.

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition.

EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging. In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects. Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology. The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

Coal-gangue sound recognition using hybrid multi-branch CNN based on attention mechanism fusion in noisy environments

The coal-gangue recognition technology plays an important role in the intelligent realization of fully mechanized caving face and the improvement of coal quality. Although great progress has been made for the coal-gangue recognition in recent years, most of them have not taken into account the impact of the complex environment of top coal caving on recognition performance. Herein, a hybrid multi-branch convolutional neural network (HMBCNN) is proposed for coal-gangue recognition, which based on improved Mel Frequency Cepstral Coefficient (MFCC) as well as Mel spectrogram, and attention mechanism. Firstly, the MFCC and its smooth feature matrix are input into each branch of one-dimensional multi-branch convolutional neural network, and the spliced features are extracted adaptively through multi-head attention mechanism. Secondly, the Mel spectrogram and its first-order derivative are input into each branch of the two-dimensional multi-branch convolutional neural network respectively, and the effective time-frequency information is paid attention to through the soft attention mechanism. Finally, at the decision-making level, the two networks are fused to establish a model for feature fusion and classification, obtaining optimal fusion strategies for different features and networks. A database of sound pressure signals under different signal-to-noise ratios and equipment operations is constructed based on a large amount of data collected in the laboratory and on-site. Comparative experiments and discussions are conducted on this database with advanced algorithms and different neural network structures. The results show that the proposed method achieves higher recognition accuracy and better robustness in noisy environments.