Multi-attention Mechanism Research Articles

AGCNAF: predicting disease-gene associations using GCN and multi-head attention to fuse the similarity features

Abstract Association prediction between diseases and genes is a critical step in revealing the molecular mechanisms of diseases and developing drug treatment strategies. With the explosive growth of data in the biomedical field, how to effectively utilize these data for accurate prediction has become a hotspot and challenge in current research. To overcome the limitations of current prediction methods in dealing with complex biological network structures and feature extraction, this study proposes AGCNAF, a method that combines an unsupervised Graph Convolutional Network (GCN) and a multi-head attention mechanism. The metagraph-guided random walk strategy enables AGCNAF to capture local and high-order topological structures in the graph, while GCN is responsible for realizing deep feature extraction of these structures. By incorporating similarity features through the multi-attention mechanism, AGCNAF achieves effective integration of global and local features, which significantly improves the prediction performance. By incorporating similarity features through the multi-attention mechanism, AGCNAF achieves effective integration of global and local features, which significantly improves the prediction performance. By utilizing the machine learning binary classification model for prediction, the experimental results through five-fold cross-validation show that AGCNAF demonstrates significant advantages in prediction performance compared to existing methods, with its AUC and AUPR reaching 0.9686 and 0.9709, respectively, and the AUC up to 0.9812 under specific conditions. 
To verify the practical application value of AGCNAF, this study also conduct case studies on Alzheimer's disease, lung cancer, and breast cancer. The results further confirm the excellent performance of AGCNAF in identifying potential disease-gene associations, which opens up new possibilities for future disease-gene research.

Early diagnosis of Alzheimer's Disease based on multi-attention mechanism.

Alzheimer's Disease is a neurodegenerative disorder, and one of its common and prominent early symptoms is language impairment. Therefore, early diagnosis of Alzheimer's Disease through speech and text information is of significant importance. However, the multimodal data is often complex and inconsistent, which leads to inadequate feature extraction. To address the problem, We propose a model for early diagnosis of Alzheimer's Disease based on multimodal attention(EDAMM). Specifically, we first evaluate and select three optimal feature extraction methods, Wav2Vec2.0, TF-IDF and Word2Vec, to extract acoustic and linguistic features. Next, by leveraging self-attention mechanism and cross-modal attention mechanisms, we generate fused features to enhance and capture the inter-modal correlation information. Finally, we concatenate the multimodal features into a composite feature vector and employ a Neural Network(NN) classifier to diagnose Alzheimer's Disease. To evaluate EDAMM, we perform experiments on two public datasets, i.e., NCMMSC2021 and ADReSSo. The results show that EDAMM improves the performance of Alzheimer's Disease diagnosis over state-of-the-art baseline approaches on both datasets.

Transformer-empowered receiver design of OFDM communication systems

With deep learning, we perform channel estimation and signal detection in massive Multiple Input Multiple Output (MIMO)-Orthogonal Frequency Division Multiplexing (OFDM) systems in this paper. Specifically, we design and extend the basic framework of receivers for MIMO-OFDM systems in an end-to-end approach. A Transformer-based MIMO-OFDM receiver called TCD-Receiver is proposed, which introduces a multi-attention mechanism to learn the channel characteristics by introducing a generic and flexible Transformer network structure. The network parameters are updated based on the relationship between the received signal and the original signal, where the final signal information is obtained without explicit channel estimation and the predicted transmit bits are directly output. The experimental results show that the TCD-Receiver proposed can effectively solve the channel distortion and detect the transmitted signals compared with the traditional communication receivers, and its performance can be comparable to that of the traditional OFDM receivers, and it also has obvious advantages in combating the complex and difficult-to-model channel environment as well as the nonlinear interference factors.

Siamese Tracking Network with Multi-attention Mechanism

Object trackers based on Siamese networks view tracking as a similarity-matching process. However, the correlation operation operates as a local linear matching process, limiting the tracker’s ability to capture the intricate nonlinear relationship between the template and search region branches. Moreover, most trackers don’t update the template and often use the first frame of an image as the initial template, which will easily lead to poor tracking performance of the algorithm when facing instances of deformation, scale variation, and occlusion of the tracking target. To this end, we propose a Simases tracking network with a multi-attention mechanism, including a template branch and a search branch. To adapt to changes in target appearance, we integrate dynamic templates and multi-attention mechanisms in the template branch to obtain more effective feature representation by fusing the features of initial templates and dynamic templates. To enhance the robustness of the tracking model, we utilize a multi-attention mechanism in the search branch that shares weights with the template branch to obtain multi-scale feature representation by fusing search region features at different scales. In addition, we design a lightweight and simple feature fusion mechanism, in which the Transformer encoder structure is utilized to fuse the information of the template area and search area, and the dynamic template is updated online based on confidence. Experimental results on publicly tracking datasets show that the proposed method achieves competitive results compared to several state-of-the-art trackers.

Detection of Crimping Defects in X-DR Images of Strain Clamps Based on Multi Attention Mechanism Fusion Network

Detection of Crimping Defects in X-DR Images of Strain Clamps Based on Multi Attention Mechanism Fusion Network

Chinese traditional painting style automatic classification based on dual-channel feature fusion with multi-attention mechanism

Existing classification models for traditional Chinese paintings mostly ignore shallow detail features, which leads to the imprecise classification of styles. To address the above problems, this paper proposes a Chinese traditional painting style automatic classification model based on dual-channel feature fusion with multi-attention mechanism. First, the spatial attention mechanism is introduced to enhance the Swin-Transformer framework to obtain the salient features of Chinese ancient painting images. Second, a dual-channel attention mechanism is constructed to extract global semantic features and local features of Chinese ancient painting images. Finally, the extracted features are fused and categorized based on the softmax classifier. To verify the feasibility and validity of the proposed model, this paper performs simulations on the Chinese painting dataset and compares it with existing algorithms.The average classification accuracy of the proposed model is 90.6[Formula: see text], with an improvement of 3.14[Formula: see text], which is better than the existing model in both visual effects and objective data comparisons.

HDMA-CGAN: Advancing Image Style Transfer with Deep Learning

The integration of artificial intelligence (AI) and deep learning heralds a transformative era in pattern recognition and computer vision, notably in image style transfer. We introduce the hierarchical dynamic multi-attention cycle generative adversarial network (HDMA-CGAN), an innovative deep learning architecture poised to redefine image style transfer capabilities. HDMA-CGAN employs a novel multi-attention mechanism and color optimization strategies, enabling precise style replication with improved fidelity and vibrancy. Our model surpasses existing benchmarks in image quality, validated by leading metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and Fréchet inception distance (FID). Although HDMA-CGAN advances the state of the art, it necessitates high computational resources and faces challenges with very high-resolution images. Future work could explore optimizing the model’s efficiency for real-time applications and extending its application to video content. This work enhances the tools available for visual content creation and digital media enhancement, leveraging advanced pattern recognition and AI techniques to significantly impact computer vision and image processing.

CrossPredGO: A Novel Light-weight Cross-modal Multi-attention Framework for Protein Function Prediction.

Proteins are represented in various ways, each contributing differently to protein-related tasks. Here, information from each representation (protein sequence, 3D structure, and interaction data) is combined for an efficient protein function prediction task. Recently, uni-modal has produced promising results with state-of-the-art attention mechanisms that learn the relative importance of features, whereas multi-modal approaches have produced promising results by simply concatenating obtained features using a computational approach from different representations which leads to an increase in the overall trainable parameters. In this paper, we propose a novel, light-weight cross-modal multi-attention (CrMoMulAtt) mechanism that captures the relative contribution of each modality with a lower number of trainable parameters. The proposed mechanism shows a higher contribution from PPI and a lower contribution from structure data. The results obtained from the proposed CrossPredGO mechanism demonstrate an increment in Fmax in the range of +(3.29 to 7.20)% with at most 31% lower trainable parameters compared with DeepGO and MultiPredGO.

Fault detection and isolation of floating wind turbine pitch system based on Kalman filter and multi-attention 1DCNN

In this paper, the fault detection and isolate problem is investigated for the pitch system of floating wind turbine. In the addressed system model, the system noises and measurement noises are correlated, and the measurement is affected by the missing phenomena. A Kalman filter is designed to handle the correlated noises and estimate the pitch angle, and a residual of the measurement of the pitch system is constructed to detection the faults. Then the fault isolation algorithm is presented based on a multi-attention mechanism one-dimensional convolutional neural network, which is employed to accurately isolate the faults. The simulation results show that the proposed method can significantly improve the accuracy of fault detection and isolation, which the fault isolation accuracy of the simulation results reaches 99.15%.

Study for lightweight finger vein recognition based on a small sample

To address several common problems of finger vein recognition, a lightweight finger vein recognition algorithm by means of a small sample has been proposed in this study. First of all, a Gabor filter is applied to deal with the images for the purpose of that these processed images can simulate a kind of situation of finger vein at low temperature, such that the generalization ability of the algorithm model can be improved as well. By cutting down the amount of convolutional layers and fully connected layers in VGG-19, a lightweight network can be given. Meanwhile, the activation function of some convolutional layers is replaced to protect the network weight that can be updated successfully. After then, a multi-attention mechanism is introduced to the modified network architecture to result in improving the ability of extracting important features. Finally, a strategy based on transfer learning has been used to reduce the training time in the model training phase. Honestly, it is obvious that the proposed finger vein recognition algorithm has a good performance in recognition accuracy, robustness and speed. The experimental results show that the recognition accuracy can arrive at about 98.45%, which has had better performance in comparison with some existing algorithms.

Terminal strip detection and recognition based on improved YOLOv7-tiny and MAH-CRNN+CTC models

For substation secondary circuit terminal strip wiring, low efficiency, less easy fault detection and inspection, and a variety of other issues, this study proposes a text detection and identification model based on improved YOLOv7-tiny and MAH-CRNN+CTC terminal lines. First, the YOLOv7-tiny target detection model is improved by the introduction of the spatially invariant multi-attention mechanism (SimAM) and the weighted bidirectional feature pyramid network (BiFPN). This also improves the feature enhancements and feature fusion ability of the model, balances various scales of characteristic information, and increases the positioning accuracy of the text test box. Then, a multi-head attention hybrid (MAH) mechanism is implemented to optimize the convolutional recurrent neural network with connectionist temporal classification (CRNN+CTC) so that the model could learn data features with larger weights and increase the recognition accuracy of the model. The findings indicate that the enhanced YOLOv7-tiny model achieves 97.39%, 98.62%, and 95.07% of precision, recall, and mean average precision (mAP), respectively, on the detection dataset. The improved MAH-CRNN+CTC model achieves 91.2% character recognition accuracy in the recognition dataset.

Mechanical Fault Diagnosis of High-Voltage Circuit Breakers with Dynamic Multi-Attention Graph Convolutional Networks Based on Adaptive Graph Construction

With the rapid development of deep learning, its powerful capabilities make it possible to perform mechanical fault diagnosis of high-voltage circuit breakers (HVCBs). Among deep learning approaches, the convolutional neural network is widely used. However, while it can extract features effectively, it also has some limitations. Specifically, it depends on a large number of training data and only takes data information into account without considering structural information. These shortcomings lead to unused information and unsatisfactory model results. To address these shortcomings, this paper proposes AKNN-DMGCN, a novel dynamic multi-attention graph convolutional network based on an adaptively constructed graph, which can achieve high accuracy and robust mechanical fault diagnosis of HVCBs. First, a novel adaptive k-nearest neighbor (AKNN) graph construction method is proposed to construct informative graphs. The AKNN method can mine the relationship between the original data samples and utilize the data and label information. Thus, it has high fault tolerance to noise signals and can construct a structure graph with rich and accurate information, which can improve the overall model performance. Then, a dynamic multi-attention graph convolutional network (DMGCN) is applied for mechanical fault diagnosis of HVCBs. DMGCN fully utilizes structural and numerical information representing HVCB signals to perform classification. DMGCN has a dynamic multi-attention mechanism with strong expressive ability, which allows it to achieve high diagnostic accuracy. The experimental results indicate that the accuracy of AKNN-DMGCN reaches 97.22% on a balanced dataset and 95.01% on an imbalanced dataset, which demonstrates that the proposed method is effective for both balanced and imbalanced samples.

A bearing surface defect detection method based on multi-attention mechanism Yolov8

Surface defects in bearings not only affect the appearance but also impact the service life and performance. Therefore, it is imperative for bearing manufacturers to conduct quality inspections before bearings leave the factory. However, traditional visual inspection methods exhibit shortcomings such as high omission rates, insufficient feature fusion and oversized models when dealing with multiple target defects in bearings. To address these challenges, this paper proposes a surface defect detection method for bearings based on an improved Yolov8 algorithm (G-Yolov8). Firstly, a C3Ghost convolutional module based on the Ghost module is constructed in YOLOv8 to simplify model computational costs. Secondly, a global attention mechanism module is designed at the end of the backbone network to increase sensitivity to implicit small target area features and optimize feature extraction efficiency. Subsequently, a deep deformable convolution feature pyramid network is constructed by introducing the deformable convolutional networks version 2 (DCNv2) and the lightweight content-aware reassembly of features upsampling operator to reduce sampling information loss and improve the fusion of multi-scale target defects. Finally, different attention mechanisms are embedded in the detection network to construct a multi-attention detection head to replace the decoupled head, refining classification and localization tasks, reducing feature confusion, and improving the model’s detection accuracy. Experimental results demonstrate that the improved algorithm achieves a 3.5% increase in mean average precision on a self-made small-scale train bearing surface defect dataset, with a 17.3% reduction in model size. This improvement not only enhances accuracy but also addresses the requirement for lightweight deployment in subsequent stages.

A Digital Animation Generation Model Based on Cycle Adversarial Neural Network

This paper proposes a digital animation generation model based on Cycle Adversarial Neural Network (CycleGAN). Compared with the classical CycleGAN, this research presents a multi-attention approach to enhance the network’s generalization. Specifically, a style enhancement module and a style cross-attention mechanism are introduced into the generator network structure, which enables the model to better parse the structural information of the content image and realize the accurate matching of content features and style features. Furthermore, the introduction of a multi-scale discriminator with a fused attention mechanism enhances the preservation of content information from the source image in the output animated image. The experiments conducted in this study demonstrate that the model proposed in this paper exhibits superior performance in generating digital animation. This model not only enhances the realism and variety of the generated effects, but also achieves significant improvements in the coherence and stability of long time-series animation. The contribution of this paper lies in the introduction of the multi-attention mechanism, which enhances the generalization of the network and is of great theoretical and practical significance for the development of the field of digital animation generation.

High-Resolution Model for Segmenting and Predicting Brain Tumor Based on Deep UNet with Multi Attention Mechanism

High-Resolution Model for Segmenting and Predicting Brain Tumor Based on Deep UNet with Multi Attention Mechanism

Segmentation of liver CT images based on weighted medical transformer model.

Deep convolutional neural networks have made significant strides in the field of medical image segmentation. Although existing convolutional structures enhance performance by leveraging local image information, they often lose the interdependence information between contexts. Therefore, the article utilizes the multi-attention mechanism of the Transformer structure to more comprehensively express relationships between contexts and introduced the Transformer network architecture into the field of medical image segmentation. Most models based on this Transformer structure typically require large datasets for training. However, in the medical field, the limited size of datasets makes training models with the Transformer structure challenging. To address this, the article propose a Weighted Medical Transformer (WMT) model that imposes low requirements on dataset quantity. The weighting mechanism in the WMT model aims to improve the issue of inaccurate relative positional coding when dealing with small medical datasets. Additionally, a coarse-grained and fine-grained segmentation mechanism is introduced, focusing on both the detailed aspects within image blocks and the boundary information connecting blocks. Experimental results on a liver dataset demonstrate that the model achieves F1 and IoU scores of 88.48% and 79.41%, respectively. Results on the MoNuSeg dataset show comparable high F1 and IoU scores of 79.58% and 66.19%, respectively. The model's accuracy surpasses that of U-Net++ and U-Net models. Compared to other models, this approach is applicable to scenarios with limited datasets, exhibiting high execution efficiency and accuracy.

Fault transfer diagnosis of rolling bearings across different devices via multi-domain information fusion and multi-kernel maximum mean discrepancy

The current deep learning-based intelligent diagnosis algorithms depend on large amounts of well-labeled data, but they may not perform well in engineering practice where the fault data available from a single device is limited. The significant difference in data distribution between different devices makes it more difficult to transfer diagnosis across devices. Therefore, aiming at the problem that the great difference in the data distribution of source and target domains in cross-device transfer diagnosis leads to the low diagnosis accuracy, this paper investigates a deep transfer network model based on multi-domain information fusion and multi-kernel maximum mean discrepancy (MK-MMD) for fault diagnosis of rolling bearings across different devices. Firstly, to address the problem that single-domain information is difficult to adequately characterize rolling bearing health status information of different devices, a multi-domain information feature extractor based on multi-attention mechanism is constructed to capture the important features of vibration signals in different transform domains. Secondly, the extracted multi-domain features are input into the bidirectional gated recurrent unit (BiGRU) network for feature fusion, and MK-MMD is used to narrow the distribution distance between the source domain and target domain to obtain domain invariant features, and then achieve the transfer diagnosis of rolling bearing faults between different devices. Finally, the method investigated is verified by rolling bearing fault transfer diagnosis tests between different devices, and the results show that the suggested method can adapt to the feature distribution between different domains, and improve the transfer diagnosis accuracy between different devices.

A robust multi-branch multi-attention-mechanism EEGNet for motor imagery BCI decoding

BackgroundMotor-Imagery-based Brain-Computer Interface (MI-BCI) is a promising technology to assist communication, movement, and neurological rehabilitation for motor-impaired individuals. Electroencephalography (EEG) decoding techniques using deep learning (DL) possess noteworthy advantages due to automatic feature extraction and end-to-end learning. However, the DL-based EEG decoding models tend to show large variations due to intersubject variability of EEG, which results from inconsistencies of different subjects’ optimal hyperparameters. New methodsThis study proposes a multi-branch multi-attention mechanism EEGNet model (MBMANet) for robust decoding. It applies the multi-branch EEGNet structure to achieve various feature extractions. Further, the different attention mechanisms introduced in each branch attain diverse adaptive weight adjustments. This combination of multi-branch and multi-attention mechanisms allows for multi-level feature fusion to provide robust decoding for different subjects. ResultsThe MBMANet model has a four-classification accuracy of 83.18% and kappa of 0.776 on the BCI Competition IV-2a dataset, which outperforms other eight CNN-based decoding models. This consistently satisfactory performance across all nine subjects indicates that the proposed model is robust. ConclusionsThe combine of multi-branch and multi-attention mechanisms empowers the DL-based models to adaptively learn different EEG features, which provides a feasible solution for dealing with data variability. It also gives the MBMANet model more accurate decoding of motion intentions and lower training costs, thus improving the MI-BCI’s utility and robustness.

Small-Scale Foreign Object Debris Detection Using Deep Learning and Dual Light Modes

The intrusion of foreign objects on airport runways during aircraft takeoff and landing poses a significant safety threat to air transportation. Small-scale Foreign Object Debris (FOD) cannot be ruled out on time by traditional manual inspection, and there is also a potential risk of secondary foreign body intrusion. A deep-learning-based intelligent detection method is proposed to solve the problem of low accuracy and low efficiency of small-scale FOD detection. Firstly, a dual light camera system is utilized for the collection of FOD data. It generates a dual light FOD dataset containing both infrared and visible light images. Subsequently, a multi-attention mechanism and a bidirectional feature pyramid are integrated into the baseline network YOLOv5. This integration prioritizes the extraction of foreign object features and boosts the network’s ability to distinguish FOD from complex backgrounds. Additionally, it enhances the fusion of higher-level features to improve the representation of multi-scale objects. To ensure fast and accurate localization and recognition of targets, the Complete-IoU (CIoU) loss function is used to optimize the bounding boxes’ positions. The experimental results indicate that the proposed model achieves a detection speed of 36.3 frame/s, satisfying real-time detection requirements. The model also attains an average accuracy of 91.1%, which is 7.4% higher than the baseline network. Consequently, this paper verifies the effectiveness and practical utility of our algorithm for the detection of small-scale FOD targets.

Joint analysis of interaction and psychological characteristics in english teaching based on multimodal integration

The intersection of psychology and English teaching is profound, as the application of psychological principles not only guides specific English instruction but also elevates the overall quality of teaching. This paper takes a multimodal approach, incorporating image, acoustics, and text information, to construct a joint analysis model for English teaching interaction and psychological characteristics. The novel addition of an attention mechanism in the multimodal fusion process enables the development of an English teaching psychological characteristics recognition model. The initial step involves balancing the proportions of each emotion, followed by achieving multimodal alignment. In the cross-modal stage, the interaction of image, acoustics, and text is facilitated through a cross-modal attention mechanism. The utilization of a multi-attention mechanism not only enhances the network’s representation capabilities but also streamlines the complexity of the model. Empirical results demonstrate the model’s proficiency in accurately identifying five psychological characteristics. The proposed method achieves a classification accuracy of 90.40% for psychological features, with a commendable accuracy of 78.47% in multimodal classification. Furthermore, the incorporation of the attention mechanism in feature fusion contributes to an improved fusion effect.