Multi-head Attention Mechanism Research Articles

Tackling heterogeneity in medical federated learning via aligning vision transformers

Federated learning enables training models on distributed, privacy-sensitive medical imaging data. However, data heterogeneity across participating institutions leads to reduced model performance and fairness issues, especially for underrepresented datasets. To address these challenges, we propose leveraging the multi-head attention mechanism in Vision Transformers to align the representations of heterogeneous data across clients. By focusing on the attention mechanism as the alignment objective, our approach aims to improve both the accuracy and fairness of federated learning models in medical imaging applications. We evaluate our method on the IQ-OTH/NCCD Lung Cancer dataset, simulating various levels of data heterogeneity using Latent Dirichlet Allocation (LDA). Our results demonstrate that our approach achieves competitive performance compared to state-of-the-art federated learning methods across different heterogeneity levels and improves the performance of models for underrepresented clients, promoting fairness in the federated learning setting. These findings highlight the potential of leveraging the multi-head attention mechanism to address the challenges of data heterogeneity in medical federated learning.

ICKA: An instruction construction and Knowledge Alignment framework for Multimodal Named Entity Recognition

Multimodal Named Entity Recognition (MNER) aims to identify entities of predefined types in text by leveraging information from multiple modalities, most notably textual and visual information. Most efforts concentrate on improving cross-modality attention mechanisms to facilitate guidance between modalities. However, they still suffer from certain limitations: (1) it is difficult to establish a unified representation to bridge the semantic gap among different modalities; (2) mining the implicit relationships between text and image is crucial yet challenging. In this paper, we propose an Instruction Construction and Knowledge Alignment Framework for MNER named ICKA to address these issues. Specifically, we first employ a multi-head cross-modal attention mechanism to obtain the cross-modal fusion representation by fusing features from text–image pairs. Then, we integrate external knowledge from the pre-trained vision-language model (VLM) to facilitate semantic alignment between text and image and obtain inter-modality connections. Next, we construct the multimodal instruction that consists of the modal features and uses the inter-modality connections as a bridge between them. We then integrate the instruction into the language model to effectively incorporate multimodal knowledge. Finally, we perform sequence labeling using a Conditional Random Fields (CRF) decoder with a gating mechanism. The proposed method achieves F1 scores of 75.42% on the Twitter2015 dataset and 87.12% on the Twitter2017 dataset, demonstrating the competitiveness of our method.

Fault Diagnosis Method for Tractor Transmission System Based on Improved Convolutional Neural Network–Bidirectional Long Short-Term Memory

In response to the problems of limited algorithms and low diagnostic accuracy for fault diagnosis in large tractor transmission systems, as well as the high noise levels in tractor working environments, a defect detection approach for tractor transmission systems is proposed using an enhanced convolutional neural network (CNN) and a bidirectional long short-term memory neural network (BILSTM). This approach uses a one-dimensional convolutional neural network (1DCNN) to create three feature extractors of varying scales, directly extracting feature information from different levels of the raw vibration signals. Simultaneously, in order to enhance the model’s predicted accuracy and learn the data features more effectively, it presents the multi-head attention mechanism (MHA). To overcome the issue of high noise levels in tractor working environments and enhance the model’s robustness, an adaptive soft threshold is introduced. Finally, to recognize and classify faults, the fused feature data are fed into a classifier made up of bidirectional long short-term memory (BILSTM) and fully linked layers. The analytical findings demonstrate that the fault recognition accuracy of the method described in this article is over 98%, and it also has better performance in noisy environments.

Prediction of protein secondary structure by the improved TCN-BiLSTM-MHA model with knowledge distillation

Secondary structure prediction is a key step in understanding protein function and biological properties and is highly important in the fields of new drug development, disease treatment, bioengineering, etc. Accurately predicting the secondary structure of proteins helps to reveal how proteins are folded and how they function in cells. The application of deep learning models in protein structure prediction is particularly important because of their ability to process complex sequence information and extract meaningful patterns and features, thus significantly improving the accuracy and efficiency of prediction. In this study, a combined model integrating an improved temporal convolutional network (TCN), bidirectional long short-term memory (BiLSTM), and a multi-head attention (MHA) mechanism is proposed to enhance the accuracy of protein prediction in both eight-state and three-state structures. One-hot encoding features and word vector representations of physicochemical properties are incorporated. A significant emphasis is placed on knowledge distillation techniques utilizing the ProtT5 pretrained model, leading to performance improvements. The improved TCN, achieved through multiscale fusion and bidirectional operations, allows for better extraction of amino acid sequence features than traditional TCN models. The model demonstrated excellent prediction performance on multiple datasets. For the TS115, CB513 and PDB (2018–2020) datasets, the prediction accuracy of the eight-state structure of the six datasets in this paper reached 88.2%, 84.9%, and 95.3%, respectively, and the prediction accuracy of the three-state structure reached 91.3%, 90.3%, and 96.8%, respectively. This study not only improves the accuracy of protein secondary structure prediction but also provides an important tool for understanding protein structure and function, which is particularly applicable to resource-constrained contexts and provides a valuable tool for understanding protein structure and function.

Cross-region feature fusion with geometrical relationship for OCR-based image captioning

Automatically generating a readable sentence that describes the text-contained image is a challenging task. Compared to traditional image captioning algorithms, OCR-based image captioning focuses on reading OCR tokens in images and understanding them with the image content to generate descriptions. However, existing research mainly concentrate on improving the quantity and quality of obtaining OCR tokens and exploring their spatial relationships while lacking investigation into how to effectively join OCR tokens with image content. This paper proposes a cross-region feature fusion with a geometrical relationship Transformer(CFGR-Transformer) for OCR-based image captioning. The network first establishes the associations between the OCR and object regions of the image by constructing relative geometric relationships, including width/height difference, distance, IOU(Intersection over Union), inclusion relationships, and angles offset, and then incorporates intra-region and cross-region features to aggregate entities from different modalities by a multi-head attention mechanism based on relative relationships. Benefiting from the guidance of the relative relationship, visual entities like OCR tokens and object regions can consider multiple relative relationships as the attention weight for feature fusion within each subspace. Extensive experiments conducted on the TextCaps dataset demonstrate the effectiveness of the proposed CFGR-Transformer method. In particular, our results on the online testing of TextCaps achieve an improvement in CIDEr score from 93.0% to 98.2%.

Prediction of Sunspot Number with Hybrid Model Based on 1D-CNN, BiLSTM and Multi-Head Attention Mechanism

Sunspots have a significant impact on human activities. In this study, we aimed to improve solar activity prediction accuracy. To predict the sunspot number based on different aspects, such as extracted features and relationships among data, we developed a hybrid model that includes a one-dimensional convolutional neural network (1D-CNN) for extracting the features of sunspots and bidirectional long short-term memory (BiLSTM) embedded with a multi-head attention mechanism (MHAM) to learn the inner relationships among data and finally predict the sunspot number. We evaluated our model and several existing models according to different evaluation indicators, such as mean absolute error (MAE) and root mean square error (RMSE). Compared with the informer, stacked LSTM, XGBoost-DL, and EMD-LSTM-AM models, the RMSE and MAE of our results were more than 42.5% and 65.1% lower, respectively. The experimental results demonstrate that our model has higher accuracy than other methods.

Quality non-destructive sorting of large yellow croaker based on image recognition

The YOLOv7 algorithm was used to establish a fast and non-destructive quality identification model for large yellow croaker (Larimichthys crocea) in this study. The cross multi-head attention mechanism in the swin transformer was incorporated into the neck of YOLOv7 architecture to enhance the recognition performance. The freshness classification model based on the total volatile basic nitrogen value was evaluated by freshness indicators, visual features, and texture profile analysis (TPA). The findings indicated that the enhanced model achieved an accuracy rate of 98.6% in freshness classification, which was higher than 85.6% of the original model. Visual features (fish-eye plumpness and turbidity) were highly correlated with all freshness indexes (all above 0.8). The accuracy of freshness discrimination in different lighting environments was also greater than 90%. These results collectively indicate the potential for the eye region images to serve as a reliable indicator for the sorting of freshness in large yellow croakers.

Dynamic Spatio-Temporal Adaptive Graph Convolutional Recurrent Networks for Vacant Parking Space Prediction

The prediction of vacant parking spaces (VPSs) can reduce the time drivers spend searching for parking, thus alleviating traffic congestion. However, previous studies have mostly focused on modeling the temporal features of VPSs using historical data, neglecting the complex and extensive spatial characteristics of different parking lots within the transportation network. This is mainly due to the lack of direct physical connections between parking lots, making it challenging to quantify the spatio-temporal features among them. To address this issue, we propose a dynamic spatio-temporal adaptive graph convolutional recursive network (DSTAGCRN) for VPS prediction. Specifically, DSTAGCRN divides VPS data into seasonal and periodic trend components and combines daily and weekly information with node embeddings using the dynamic parameter-learning module (DPLM) to generate dynamic graphs. Then, by integrating gated recurrent units (GRUs) with the parameter-learning graph convolutional recursive module (PLGCRM) of DPLM, we infer the spatio-temporal dependencies for each time step. Furthermore, we introduce a multihead attention mechanism to effectively capture and fuse the spatio-temporal dependencies and dynamic changes in the VPS data, thereby enhancing the prediction performance. Finally, we evaluate the proposed DSTAGCRN on three real parking datasets. Extensive experiments and analyses demonstrate that the DSTAGCRN model proposed in this study not only improves the prediction accuracy but can also better extract the dynamic spatio-temporal characteristics of available parking space data in multiple parking lots.

Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting.

Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features associated with the corrupted regions are degraded and incur inaccurate self attention. This problem, termed query degradation, may be mitigated by first completing optical flows and then using the flows to guide the self attention, which was verified in our previous work - flow-guided transformer (FGT). We further exploit the flow guidance and propose FGT++ to pursue more effective and efficient video inpainting. First, we design a lightweight flow completion network by using local aggregation and edge loss. Second, to address the query degradation, we propose a flow guidance feature integration module, which uses the motion discrepancy to enhance the features, together with a flow-guided feature propagation module that warps the features according to the flows. Third, we decouple the transformer along the temporal and spatial dimensions, where flows are used to select the tokens through a temporally deformable MHSA mechanism, and global tokens are combined with the inner-window local tokens through a dual-perspective MHSA mechanism. FGT++ is experimentally evaluated to be outperforming the existing video inpainting networks qualitatively and quantitatively.

Temporal graph convolutional network for multi-agent reinforcement learning of action detection

Most action detection techniques process untrimmed action videos using temporal context, without explicitly consider the inherent spatio-temporal context information, leading to limited accuracy in the cases with big spatial complexity and temporal redundancy. To this end, by intuitively dealing with the problem following a two-stage ”clip selection + clip classification” scheme, this paper proposes to formulate action detection as a Markov process and builds up a multi-agent reinforcement learning framework capturing global structural relationships of videos to optimize the selection and classification, simultaneously and progressively. In particular, a temporal graph convolutional network is constructed to represent the spatio-temporal correlations of video clips, which are initialized by evenly sampling, and further adjusted via learning the rewards adaptively for multi-agent cooperation. Multi-head dot-product attention mechanism is adopted to integrate the relations of latent CNN features of interacting agents. Our framework is jointly learnt by fusing the objectives of clip selection policy and clip recognition. The proposed method comprises a novel graph convolutional network based spatio-temporal semantic observation module which captures topological features among nearby agents, and a new policy module that segments actions according to the rewards from the objectives of action recognition. Extensive experiments are conducted on ActivityNet v1.3 and THUMOS14, with 30.13% and 55.4% mAP obtained, demonstrate the applicability and superiority of our approach.

Advancing architectural heritage: precision decoding of East Asian timber structures from Tang dynasty to traditional Japan

The convergence of cultural and aesthetic elements in timber structures from China’s Tang Dynasty (618–907 AD) and traditional Japanese architecture provides a rich tapestry of architectural evolution and cross-cultural exchanges. Addressing the challenge of distinguishing and understanding the intricate styles of these structures is significant for both historical comprehension and preservation efforts. This research introduces an innovative approach by integrating the Multi-Head Attention (MHA) mechanism into the YOLOv8 model, enhancing the detection of architectural features with improved precision and recall. Our novel YOLOv8-MHA model not only demonstrates a notable improvement in recognizing intricate architectural details but also significantly advances the state of the art in object detection within complex settings. Quantitative results underscore the model’s effectiveness, achieving a precision of 95.6%, a recall of 85.6%, and a mean Average Precision (mAP@50) of 94% across various Intersection over Union (IoU) thresholds. These metrics highlight the model’s superior capability to accurately identify and classify architectural elements, especially within environments rich with nuanced details, utilizing the enhanced YOLOv8-MHA algorithm. The application of our model extends beyond mere architectural analysis; it offers new insights into the intricate interplay of cultural identity and adaptability inherent in East Asian architectural heritage. The study establishes a solid foundation for the meticulous classification and analysis of architectural styles in timber structures within an expansive cultural and historical context, thereby enriching our understanding and preservation of these traditions.

Design of Tool Wear Monitoring System in Bone Material Drilling Process

Biological bone materials, complex and anisotropic, require precise machining in surgeries. Bone drilling, a key technique, is susceptible to increased friction from tool wear, leading to excessive forces and high temperatures that can damage bone and surrounding tissues, affecting recovery. This study develops a monitoring platform to assess tool wear during bone drilling, employing an experimental setup that gathers triaxial force and vibration data. A recognition model using a bidirectional long short-term memory network (BI-LSTM) with a multi-head attention mechanism identified wear levels. This model, termed ABI-LSTM, was optimized and benchmarked against SVR, RNN, and CNN models. The results from implementing the ABI-LSTM-based monitoring system demonstrated its efficacy in detecting tool wear, thereby potentially reducing surgical risks such as osteonecrosis and drill breakage, and enhancing surgical outcomes.

EEG Emotion Recognition Employing RGPCN-BiGRUAM: ReliefF-Based Graph Pooling Convolutional Network and BiGRU Attention Mechanism

Emotion recognition plays a crucial role in affective computing, and electroencephalography (EEG) signals are increasingly applied in this field due to their effectiveness in reflecting brain activity. In this paper, we propose a novel EEG emotion recognition model that combines the ReliefF-based Graph Pooling Convolutional Network and BiGRU Attention Mechanisms (RGPCN-BiGRUAM). RGPCN-BiGRUAM effectively integrates the advantages of graph convolutional networks and recurrent neural networks. By incorporating ReliefF weights and an attention mechanism into graph pooling, our model enhances the aggregation of high-quality features while discarding irrelevant ones, thereby improving the efficiency of information transmission. The implementation of a multi-head attention mechanism fusion in BiGRU addresses the limitations of single-output features, achieving optimal selection of global features. Comparative experiments on public datasets SEED and DEAP demonstrate that our proposed RGPCN-BiGRUAM significantly improves classification performance compared to classic algorithms, achieving state-of-the-art results. Ablation studies further validate the design principles of our model. The results of this study indicate that RGPCN-BiGRUAM has strong potential for EEG emotion recognition, offering substantial possibilities for future applications.

Enhanced coalbed methane well production prediction framework utilizing the CNN-BL-MHA approach

As the mechanization of the CBM extraction process advances and geological conditions continuously evolve, the production data from CBM wells is deviating increasingly from linearity, thereby presenting a significant challenge in accurately predicting future gas production from these wells. When it comes to predicting the production of CBM, a single deep-learning model can face several drawbacks such as overfitting, gradient explosion, and gradient disappearance. These issues can ultimately result in insufficient prediction accuracy, making it important to carefully consider the limitations of any given model. It’s impressive to see how advanced technology can enhance the prediction accuracy of CBM. In this paper, the use of a CNN model to extract features from CBM well data and combine it with Bi-LSTM and a Multi-Head Attention mechanism to construct a production prediction model for CBM wells—the CNN-BL-MHA model—is fascinating. It is even more exciting that predictions of gas production for experimental wells can be conducted using production data from Wells W1 and W2 as the model’s database. We compared and analyzed the prediction results obtained from the CNN-BL-MHA model we constructed with those from single models like ARIMA, LSTM, MLP, and GRU. The results show that the CNN-BL-MHA model proposed in the study has shown promising results in improving the accuracy of gas production prediction for CBM wells. It’s also impressive that this model demonstrated super stability, which is essential for reliable predictions. Compared to the single deep learning model used in this study, its prediction accuracy can be improved up to 35%, and the prediction results match the actual yield data with lower error.

U-HRNet+: a real-time precipitation estimation method based on the fusion of temporal and spatial domain features

ABSTRACT The precipitation estimation with high spatial and temporal resolution over Tibet is of great significance for monitoring the stability of the plateau ecosystem and early warning of natural disasters. In response to the problem of the unsatisfactory accuracy in traditional real-time precipitation inversion methods and quantitative accuracy estimation products in this region. This paper adopts deep-learning methods to explore the corresponding relationship between precipitation and multi-band observation data from geostationary satellite imagery. The U-High Resolution Network (U-HRNet) as backbone is fully exploited to generate multi-scale features and fused using attention gates to extract high-resolution spatial representations representing precipitation clouds, thereby achieving precipitation estimation of target clouds at different scales. In addition, in response to the characteristics of high precipitation intensity and drastic cloud cluster changes in summer convective weather within a short period of time, a multi-head attention mechanism is introduced to extract temporal information from time series inputs and fuse it with spatial features to improve the accuracy of real-time precipitation estimation. Compared with the operational precipitation products and baseline deep-learning models, the classification metrics such as Probability of detection (POD), False Alarm Ratio (FAR) and Critical success index (CSI) show that the proposed model (U-HRNet+) has better performance in test set and has high generalization ability for estimating short-term heavy rainfall over Tibet and its surrounding areas in summer. Therefore, the proposed model has the potential to serve as a more accurate and reliable satellite-based precipitation estimation product.

Multi-domain fusion for cargo UAV fault diagnosis knowledge graph construction

The fault diagnosis of cargo UAVs (Unmanned Aerial Vehicles) is crucial to ensure the safety of logistics distribution. In the context of smart logistics, the new trend of utilizing knowledge graph (KG) for fault diagnosis is gradually emerging, bringing new opportunities to improve the efficiency and accuracy of fault diagnosis in the era of Industry 4.0. The operating environment of cargo UAVs is complex, and their faults are typically closely related to it. However, the available data only considers faults and maintenance data, making it difficult to diagnose faults accurately. Moreover, the existing KG suffers from the problem of confusing entity boundaries during the extraction process, which leads to lower extraction efficiency. Therefore, a fault diagnosis knowledge graph (FDKG) for cargo UAVs constructed based on multi-domain fusion and incorporating an attention mechanism is proposed. Firstly, the multi-domain ontology modeling is realized based on the multi-domain fault diagnosis concept analysis expression model and multi-dimensional similarity calculation method for cargo UAVs. Secondly, a multi-head attention mechanism is added to the BERT-BILSTM-CRF network model for entity extraction, relationship extraction is performed through ERNIE, and the extracted triples are stored in the Neo4j graph database. Finally, the DJI cargo UAV failure is taken as an example for validation, and the results show that the new model based on multi-domain fusion data is better than the traditional model, and the precision rate, recall rate, and F1 value can reach 87.52%, 90.47%, and 88.97%, respectively.

EPI-Trans: an effective transformer-based deep learning model for enhancer promoter interaction prediction

BackgroundRecognition of enhancer–promoter Interactions (EPIs) is crucial for human development. EPIs in the genome play a key role in regulating transcription. However, experimental approaches for classifying EPIs are too expensive in terms of effort, time, and resources. Therefore, more and more studies are being done on developing computational techniques, particularly using deep learning and other machine learning techniques, to address such problems. Unfortunately, the majority of current computational methods are based on convolutional neural networks, recurrent neural networks, or a combination of them, which don’t take into consideration contextual details and the long-range interactions between the enhancer and promoter sequences. A new transformer-based model called EPI-Trans is presented in this study to overcome the aforementioned limitations. The multi-head attention mechanism in the transformer model automatically learns features that represent the long interrelationships between enhancer and promoter sequences. Furthermore, a generic model is created with transferability that can be utilized as a pre-trained model for various cell lines. Moreover, the parameters of the generic model are fine-tuned using a particular cell line dataset to improve performance.ResultsBased on the results obtained from six benchmark cell lines, the average AUROC for the specific, generic, and best models is 94.2%, 95%, and 95.7%, while the average AUPR is 80.5%, 66.1%, and 79.6% respectively.ConclusionsThis study proposed a transformer-based deep learning model for EPI prediction. The comparative results on certain cell lines show that EPI-Trans outperforms other cutting-edge techniques and can provide superior performance on the challenge of recognizing EPI.

Enhanced Thermal Modeling of Electric Vehicle Motors Using a Multihead Attention Mechanism

The rapid advancement of electric vehicles (EVs) accentuates the criticality of efficient thermal management systems for electric motors, which are pivotal for performance, reliability, and longevity. Traditional thermal modeling techniques often struggle with the dynamic and complex nature of EV operations, leading to inaccuracies in temperature prediction and management. This study introduces a novel thermal modeling approach that utilizes a multihead attention mechanism, aiming to significantly enhance the prediction accuracy of motor temperature under varying operational conditions. Through meticulous feature engineering and the deployment of advanced data handling techniques, we developed a model that adeptly navigates the intricacies of temperature fluctuations, thereby contributing to the optimization of EV performance and reliability. Our evaluation using a comprehensive dataset encompassing temperature data from 100 electric vehicles illustrates our model’s superior predictive performance, notably improving temperature prediction accuracy.

Accurate prediction of drug combination risk levels based on relational graph convolutional network and multi-head attention

BackgroundAccurately identifying the risk level of drug combinations is of great significance in investigating the mechanisms of combination medication and adverse reactions. Most existing methods can only predict whether there is an interaction between two drugs, but cannot directly determine their accurate risk level.MethodsIn this study, we propose a multi-class drug combination risk prediction model named AERGCN-DDI, utilizing a relational graph convolutional network with a multi-head attention mechanism. Drug-drug interaction events with varying risk levels are modeled as a heterogeneous information graph. Attribute features of drug nodes and links are learned based on compound chemical structure information. Finally, the AERGCN-DDI model is proposed to predict drug combination risk level based on heterogenous graph neural network and multi-head attention modules.ResultsTo evaluate the effectiveness of the proposed method, five-fold cross-validation and ablation study were conducted. Furthermore, we compared its predictive performance with baseline models and other state-of-the-art methods on two benchmark datasets. Empirical studies demonstrated the superior performances of AERGCN-DDI.ConclusionsAERGCN-DDI emerges as a valuable tool for predicting the risk levels of drug combinations, thereby aiding in clinical medication decision-making, mitigating severe drug side effects, and enhancing patient clinical prognosis.

A short-term power load forecasting system based on data decomposition, deep learning and weighted linear error correction with feedback mechanism

Accurate power load forecasting enables Independent System Operators (ISOs) to precisely quantify the demand patterns of users and achieve efficient management of the smart grid. However, with the increasing variety of power consumption patterns, the power load data displays increasingly irregular characteristics, which posing great challenges for accurate load forecasting. In order to solve above problem, a novel power load forecasting system is proposed based on data denoising, customized deep learning and weighted linear error correction. Specifically, we first proposed an improved optimization algorithm IGWO-JAYA which enhanced the Grey Wolf Optimizer (GWO) algorithm by using Halton low-discrepancy sequence and the mechanism of JAYA algorithm. In data denoising, the proposed optimizer was employed to optimize the Variational Mode Decomposition (VMD), enabling data-driven intelligent denoising. The customized deep learning framework contained multi-layer Convolution Neural Network (CNN), Bi-directional Long Short-Term Memory (Bi-LSTM) and Multi-Head Attention mechanism. The effective integration of these layers can significantly improve the capacity for nonlinear fitting of deep learning. In weighted linear error correction, the IGWO-JAYA algorithm was employed to determine the appropriate weight for point forecasting values and residual forecasting values. By weighting them, the forecasting precision has been further enhanced. To verify the forecasting ability of the system, we conducted experiments on power load datasets from four states in Australia and found that it has the best performance compared with all rivals. In the discussion, we demonstrated the convergence efficiency of the IGWO-JAYA algorithm by CEC test function.