Multi-head Self-attention Research Articles

Personalized multi-head self-attention network for news recommendation

With the rapid explosion of online news and user population, personalized news recommender systems have proved to be efficient ways of alleviating information overload problems by suggesting information which attracts users in line with their tastes. Exploring relationships among words and news is critical to structurally model users’ latent tastes including interested domains, while selecting informative words and news can directly reflect users’ interests. Most of the current studies do not provide an effective framework that combines distilling users’ interested latent spaces and explicit points systematically. Moreover, introducing more advanced techniques to merely chase accuracy has become a universal phenomenon. In this study, we design a Personalized Multi-Head Self-Attention Network (PMSN) for news recommendation, which combines multi-head self-attention network with personalized attention mechanism from both word and news levels. Multi-head self-attention mechanism is used to model interactions among words and news, exploring latent interests. Personalized attention mechanism is applied by embedding users’ IDs to highlight informative words and news, which can enhance the interpretability of personalization. Comprehensive experiments conducted using two real-world datasets demonstrate that PMSN efficiently outperforms state-of-the-art methods in terms of recommendation accuracy, without complicated structure design and exhausted even external resources consumption. Furthermore, visualized case study validates that attention mechanism indeed increases the interpretability.

Training-Free Transformer Architecture Search With Zero-Cost Proxy Guided Evolution.

Transformers have shown remarkable performance, however, their architecture design is a time-consuming process that demands expertise and trial-and-error. Thus, it is worthwhile to investigate efficient methods for automatically searching high-performance Transformers via Transformer Architecture Search (TAS). In order to improve the search efficiency, training-free proxy based methods have been widely adopted in Neural Architecture Search (NAS). Whereas, these proxies have been found to be inadequate in generalizing well to Transformer search spaces, as confirmed by several studies and our own experiments. This paper presents an effective scheme for TAS called TRansformer Architecture search with ZerO-cost pRoxy guided evolution (T-Razor) that achieves exceptional efficiency. First, through theoretical analysis, we discover that the synaptic diversity of multi-head self-attention (MSA) and the saliency of multi-layer perceptron (MLP) are correlated with the performance of corresponding Transformers. The properties of synaptic diversity and synaptic saliency motivate us to introduce the ranks of synaptic diversity and saliency that denoted as DSS++ for evaluating and ranking Transformers. DSS++ incorporates correlation information among sampled Transformers to provide unified scores for both synaptic diversity and synaptic saliency. We then propose a block-wise evolution search guided by DSS++ to find optimal Transformers. DSS++ determines the positions for mutation and crossover, enhancing the exploration ability. Experimental results demonstrate that our T-Razor performs competitively against the state-of-the-art manually or automatically designed Transformer architectures across four popular Transformer search spaces. Significantly, T-Razor improves the searching efficiency across different Transformer search spaces, e.g., reducing required GPU days from more than 24 to less than 0.4 and outperforming existing zero-cost approaches. We also apply T-Razor to the BERT search space and find that the searched Transformers achieve competitive GLUE results on several Neural Language Processing (NLP) datasets. This work provides insights into training-free TAS, revealing the usefulness of evaluating Transformers based on the properties of their different blocks.

Provenance-based APT campaigns detection via masked graph representation learning

Advanced Persistent Threats (APTs) are well-planned, persistent, and highly stealthy cyberattacks designed to steal confidential information or disrupt specific target systems. Recent studies have used system audit logs to construct provenance graphs that describe system interactions to detect potentially malicious activities. Although they are effective, they still suffer from problems such as the need for a priori knowledge, lack of attack data, and high computational overhead that limit their application. In this paper, we propose a self-supervised learning-based APT detection model, APT-MGL, which learns the embedded representations of nodes through a graph mask self-encoder and transforms the detection problem into an outlier detection problem for malicious nodes. APT-MGL characterizes the behavior of nodes based on node type, action, and interaction frequency, and fuses the features through a multi-head self-attention mechanism. Then the node embedding is obtained by combining graph features and structural information using masked graph representation learning. Finally, the unsupervised outlier detection method is used to analyze the computed embeddings and obtain the final detection results. The experimental results show that APT-MGL outperforms existing monitoring models and achieves a small overhead.

ProTformer: Transformer-based model for superior prediction of protein content in lablab bean (Lablab purpureus L.) using Near-Infrared Reflectance spectroscopy

Lablab bean (Lablab purpureus L.), known for its higher protein content provides a promising alternative to reduce reliance on animal-based proteins and support sustainable agriculture. Nowadays, traditional methods for nutritional profiling have been outperformed by Convolutional Neural Networks (CNNs) integrated with Near-Infrared Reflectance Spectroscopy (NIRS). However, an advanced technique called Transformers, despite their potential, remains underexplored for prediction of key quality traits of crops. Thus, in the present study, we aimed to predict lablab bean protein content using a Transformer-based model (termed ProTformer) coupled with NIRS. The Dumas combustion method was used to determine protein content from 112 lablab bean genotypes. The performance of this model was compared with 1D CNN and Modified Partial Least Squares (MPLS)-based models. These models were evaluated using the Coefficient of Determination (RSQ), Residual Prediction Deviation (RPD), bias, and Corrected Standard Error of Prediction (SEP(C)). Our findings show that the Transformer-based model achieved the highest predictive accuracy (RSQ = 0.977, RPD = 13.92), surpassing the 1D CNN (RSQ = 0.956, RPD = 11.33) and MPLS (RSQ = 0.943, RPD = 10.53)- based models by 22.86% and 32.21% in RPD, respectively. The multi-head self-attention mechanism present in the Transformer-based model enables it to concurrently focus on different spectral regions, resulting in superior predictive performance. This study shows the potential of using Transformer-based models coupled with NIRS for nutritional profiling which could be effectively adapted for other crops for rapid screening of large germplasm available at global repositories.

A high-resolution method for direction of arrival estimation based on an improved self-attention module.

The high-resolution direction of arrival (DOA) estimation is a prominent research issue in underwater acoustics. The existing high-resolution methods include subspace methods and sparse representation methods. However, the performance of subspace methods suffers from low signal-to-noise ratio (SNR) and limited snapshots conditions, and the computational complexity of sparse representation methods is too high. The neural network methods are emerging high-resolution methods. However, insufficient support for big data is frequently observed in underwater acoustics, and conventional network structures present challenges in further enhancing performance. To address the aforementioned problems, we propose a neural network method based on an improved self-attention module to achieve high accuracy and robust DOA estimation. First, we design a multi-head self-attention module with large-scale convolutional kernels and residual structures to improve the estimated accuracy. Second, we propose an improved input feature to enhance the robustness to non-uniform noise and unequal-intensity targets. The simulations demonstrate that the proposed method exhibits superior angle resolution compared to sparse representation methods under the same simulation conditions. The proposed method demonstrates exceptional accuracy and robustness in DOA estimation under challenging conditions of low SNR, limited snapshots, and unequal-intensity targets. The experimental results further prove the effectiveness of the proposed method.

DBCvT: Double Branch Convolutional Transformer for Medical Image Classification

Convolutional Neural Networks (CNNs) are extensively utilized in medical disease diagnosis, demonstrating the prominent performance in most cases. However, medical image processing based on deep learning faces some challenges. The limited availability and time-consuming annotations of medical image data restrict the scale and accuracy of model training. Data diversity and complexity further complicate these challenges. In order to address these issues, we introduce the Double Branch Convolutional Transformer (DBCvT), a hybrid CNN-Transformer feature extractor, which can better capture diverse fine-grained features and remain suitable for small datasets. In this model, separable downsampling convolution (SDConv) is used to mitigate excessive information loss during downsampling in standard convolutions. Additionally, we propose the Dual branch Channel Efficient multi-head Self-Attention (DCESA) mechanism to enhance the self-attention efficiency, consequently elevating network performance and effectiveness. Moreover, we introduce a novel convolutional channel-enhanced Attention mechanism to strengthen inter-channel relationships within feature maps post self-attention. The experiments of DBCvT on various medical image datasets have demonstrated the outstanding classification performance and generalization capability of the proposed model.

A2HTL: An Automated Hybrid Transformer-Based Learning for Predicting Survival of Esophageal Cancer Using CT Images.

Esophageal cancer is a common malignant tumor, precisely predicting survival of esophageal cancer is crucial for personalized treatment. However, current region of interest (ROI) based methodologies not only necessitate prior medical knowledge for tumor delineation, but may also cause the model to be overly sensitive to ROI. To address these challenges, we develop an automated Hybrid Transformer based learning that integrates a Hybrid Transformer size-aware U-Net with a ranked survival prediction network to enable automatic survival prediction for esophageal cancer. Specifically, we first incorporate the Transformer with shifted windowing multi-head self-attention mechanism (SW-MSA) into the base of the U-Net encoder to capture the long-range dependency in CT images. Furthermore, to alleviate the imbalance between the ROI and the background in CT images, we devise a size-aware coefficient for the segmentation loss. Finally, we also design a ranked pair sorting loss to more comprehensively capture the ranked information inherent in CT images. We evaluate our proposed method on a dataset comprising 759 samples with esophageal cancer. Experimental results demonstrate the superior performance of our proposed method in survival prediction, even without ROI ground truth.

Tumor detection in breast cancer pathology patches using a Multi-scale Multi-head Self-attention Ensemble Network on Whole Slide Images

Breast cancer (BC) is the most common type of cancer among women globally and is one of the leading causes of cancer-related deaths among women. In the diagnosis of BC, histopathological assessment is the gold standard, where automated tumor detection technologies play a pivotal role. Utilizing Convolutional Neural Networks (CNNs) for automated analysis of image patches from Whole Slide Images (WSIs) enhances detection accuracy and alleviates the workload of pathologists. However, CNNs often face limitations in handling pathological patches due to a lack of sufficient contextual information and limited feature generation capabilities. To address this, we propose a novel Multi-scale Multi-head Self-attention Ensemble Network (MMSEN), which integrates a multi-scale feature generation module, a convolutional self-attention module, and an adaptive feature integration and output module, effectively optimizing the performance of classical CNNs. The design of MMSEN optimizes the capture of key information and the comprehensive integration of features in WSIs pathological patches, significantly enhancing the precision of tumor detection. Validation results from a five-fold cross-validation experiment on the PatchCamelyon (PCam) dataset demonstrate that MMSEN achieves a ROC-AUC of 99.01%±0.02%, an F1 Score of 98.00%±0.08%, a Balanced Accuracy (B-Acc) of 98.00%±0.08%, and a Matthews Correlation Coefficient (MCC) of 96.00%±0.16% (p<0.05). These results demonstrate the effectiveness and potential of MMSEN in detecting tumors from pathological patches in WSIs for BC.

MSCL-Attention: A Multi-Scale Convolutional Long Short-Term Memory (LSTM) Attention Network for Predicting CO2 Emissions from Vehicles

The transportation industry is one of the major sources of energy consumption and CO2 emissions, and these emissions have been increasing year by year. Vehicle exhaust emissions have had serious impacts on air quality and global climate change, with CO2 emissions being one of the primary causes of global warming. In order to accurately predict the CO2 emission level of automobiles, an MSCL-Attention model based on a multi-scale convolutional neural network, long short-term memory network and multi-head self-attention mechanism is proposed in this study. By combining multi-scale feature extraction, temporal sequence dependency processing, and the self-attention mechanism, the model enhances the prediction accuracy and robustness. In our experiments, the MSCL-Attention model is benchmarked against the latest state-of-the-art models in the field. The results indicate that the MSCL-Attention model demonstrates superior performance in the task of CO2 emission prediction, surpassing the leading models currently available. This study provides a new method for predicting vehicle exhaust emissions, with significant application prospects, and is expected to contribute to reducing global vehicle emissions, improving air quality, and addressing climate change.

Multi-Modal Fusion Network with Multi-Head Self-Attention for Injection Training Evaluation in Medical Education

The COVID-19 pandemic has significantly disrupted traditional medical training, particularly in critical areas such as the injection process, which require expert supervision. To address the challenges posed by reduced face-to-face interactions, this study introduces a multi-modal fusion network designed to evaluate the timing and motion aspects of the injection training process in medical education. The proposed framework integrates 3D reconstructed data and 2D images of hand movements during the injection process. The 3D data are preprocessed and encoded by a Long Short-Term Memory (LSTM) network to extract temporal features, while a Convolutional Neural Network (CNN) processes the 2D images to capture detailed image features. These encoded features are then fused and refined through a proposed multi-head self-attention module, which enhances the model’s ability to capture and weigh important temporal and image dynamics in the injection process. The final classification of the injection process is conducted by a classifier module. The model’s performance was rigorously evaluated using video data from 255 subjects with assessments made by professional physicians according to the Objective Structured Assessment of Technical Skill—Global Rating Score (OSATS-GRS)[B] criteria for time and motion evaluation. The experimental results demonstrate that the proposed data fusion model achieves an accuracy of 0.7238, an F1-score of 0.7060, a precision of 0.7339, a recall of 0.7238, and an AUC of 0.8343. These findings highlight the model’s potential as an effective tool for providing objective feedback in medical injection training, offering a scalable solution for the post-pandemic evolution of medical education.

SparseMorph: A weakly-supervised lightweight sparse transformer for mono- and multi-modal deformable image registration

PurposeDeformable image registration (DIR) is crucial for improving the precision of clinical diagnosis. Recent Transformer-based DIR methods have shown promising performance by capturing long-range dependencies. Nevertheless, these methods still grapple with high computational complexity. This work aims to enhance the performance of DIR in both computational efficiency and registration accuracy. MethodsWe proposed a weakly-supervised lightweight Transformer model, named SparseMorph. To reduce computational complexity without compromising the representative feature capture ability, we designed a sparse multi-head self-attention (SMHA) mechanism. To accumulate representative features while preserving high computational efficiency, we constructed a multi-branch multi-layer perception (MMLP) module. Additionally, we developed an anatomically-constrained weakly-supervised strategy to guide the alignment of regions-of-interest in mono- and multi-modal images. ResultsWe assessed SparseMorph in terms of registration accuracy and computational complexity.Within the mono-modal brain datasets IXI and OASIS, our SparseMorph outperforms the state-of-the-art method TransMatch with improvements of 3.2 % and 2.9 % in DSC scores for MRI-to-CT registration tasks, respectively. Moreover, in the multi-modal cardiac dataset MMWHS, our SparseMorph shows DSC score improvements of 9.7 % and 11.4 % compared to TransMatch in MRI-to-CT and CT-to-MRI registration tasks, respectively. Notably, SparseMorph attains these performance advantages while utilizing 33.33 % of the parameters of TransMatch. ConclusionsThe proposed weakly-supervised deformable image registration model, SparseMorph, demonstrates efficiency in both mono- and multi-modal registration tasks, exhibiting superior performance compared to state-of-the-art algorithms, and establishing an effective DIR method for clinical applications.

SMLS-YOLO: an extremely lightweight pathological myopia instance segmentation method.

Pathological myopia is a major cause of blindness among people under 50 years old and can result in severe vision loss in extreme cases. Currently, its detection primarily relies on manual methods, which are slow and heavily dependent on the expertise of physicians, making them impractical for large-scale screening. To tackle these challenges, we propose SMLS-YOLO, an instance segmentation method based on YOLOv8n-seg. Designed for efficiency in large-scale screenings, SMLS-YOLO employs an extremely lightweight model. First, StarNet is introduced as the backbone of SMLS-YOLO to extract image features. Subsequently, the StarBlock from StarNet is utilized to enhance the C2f, resulting in the creation of the C2f-Star feature extraction module. Furthermore, shared convolution and scale reduction strategies are employed to optimize the segmentation head for a more lightweight design. Lastly, the model incorporates the Multi-Head Self-Attention (MHSA) mechanism following the backbone to further refine the feature extraction process. Experimental results on the pathological myopia dataset demonstrate that SMLS-YOLO outperforms the baseline YOLOv8n-seg by reducing model parameters by 46.9%, increasing Box mAP@0.5 by 2.4%, and enhancing Mask mAP@0.5 by 4%. Furthermore, when compared to other advanced instance segmentation and semantic segmentation algorithms, SMLS-YOLO also maintains a leading position, suggesting that SMLS-YOLO has promising applications in the segmentation of pathological myopia images.

An effective CNN-MHSA method for the fault diagnosis of ZPW-2000A track circuit

The mainstream methods for fault diagnosis of ZPW-2000A track circuits are overly complex in extracting features from sequence data, and their feature extraction capability is relatively limited, thus impacting the performance of fault diagnosis. To address this issue, we propose a fault diagnosis model for track circuits, named as convolutional neural network incorporating multi-head self-attention mechanism (CNN-MHSA). On one hand, the local features of sequence data are locally sensed and extracted by the convolutional layer; on the other hand, the global features of sequence data are captured by establishing long-distance dependency relationships through the multi-head self-attention layer. The dual extraction of local and global features enables accurate diagnosis of faults. Finally, we collect 27 fault modes and 2 normal operation modes in the simulation system and conduct a large number of numerical experiments. The experimental results show that the fault diagnosis accuracy of the CNN-MHSA model in this paper can reach 97.45%, which is an improvement of 0.64%, 0.44%, 0.44%, 0.44%, 0.33%, 0.22%, and 0.11% compared with models such as Decision Tree, Random Forest, CatBoost, long short-term memory (LSTM), convolutional neural network (CNN), and CNN-LSTM. It also has obvious advantages in evaluation metrics such as precision, recall, Macro-F1, and Micro-F1, as well as other evaluation metrics. Therefore, the model significantly improves the comprehensive performance of ZPW-2000A track circuit fault diagnosis and can be used as a more effective fault diagnosis method.

Autonomous collision avoidance decision-making method for USV based on ATL-TD3 algorithm

To address the issue of decision-making for ship collision avoidance in complex encounter situations, an Autonomous Collision Avoidance Decision-making (ACAD) model based on Attention Long Short-Term Memory Twin Delayed Deep Deterministic Policy Gradient (ATL-TD3) algorithm for Unmanned Surface Vessel (USV) is proposed in this study. The novelty of this method is: (1) Incorporating LSTM and multi-head self-attention mechanism into the existing TD3 algorithm network enhances its focus on pertinent historical state information. Furthermore, the generalization ability of the trained model is further enhanced through optimization of the accumulated experience. (2) A comprehensive reward mechanism is designed, incorporating ship domain, arena, yaw, and other relevant factors, which solve the encounters defined in clauses 13–17 in Chapter 2 of COLREGs and immediate danger. The experiment involved a comparative analysis of the average reward curve and success rate during training under different environments. Additionally, 20 cases of Imazu classic encounters and real navigation data from the Suez Canal were utilized to validate the proposed algorithm. It is highlight that the ACAD model trained by the ATL-TD3 algorithm exhibits remarkable performance in the generalization verification, surpassing conventional collision avoidance methods.

Advancing semantic segmentation of two-dimensional materials using a semantic-adaptive transformer model

Accurate detection and characterization of two-dimensional (2D) materials are essential for their effective utilization in various applications. Traditional techniques, such as chemical vapor deposition, often produce materials with high defect density, while mechanical exfoliation is hindered by its labor-intensive and time-consuming nature. In this Letter, we propose a semantic-adaptive transformer model, termed Semptive, designed specifically for the precise detection of monolayer MoS2. Our approach integrates a semantic adaptation module with a multi-head self-attention mechanism, incorporating deep supervision and leveraging prior knowledge to enhance model performance. The model was trained on a dataset obtained through mechanical exfoliation and validated using photoluminescence spectroscopy. The experimental results reveal that Semptive significantly enhances segmentation performance compared to conventional models, achieving higher Intersection over Union and Dice scores while reducing computational demands. This method represents a notable advancement in the efficient and precise identification of 2D materials, providing substantial improvements for material characterization and device fabrication processes.

Classification of hyperspectral and LiDAR data by transformer-based enhancement

ABSTRACT The integration of multi-modal data allows for a more accurate representation of the ground characteristics. For a comprehensive interpretation of remote sensing data, existing multi-modal data fusion research mainly focuses on the joint utilization of 3D Light Detection and Ranging (LiDAR) and 2D Hyperspectral Image (HSI) data. However, existing algorithms do not pay much attention to the interaction of high-level semantic information between different modal data before fusion. This paper proposes a novel multi-modal data fusion deep learning network with the Cross-Modal Self-Attentive Feature Fusion Transformer (SAFFT). The framework employs a multi-head self-attention layer to fuse various attention information from multiple heads, effectively enhancing advanced feature information from different modalities for comprehensive integration. Experimental results on the Houston 2013 dataset demonstrate the effectiveness of the proposed method, which achieves an overall accuracy (OA) of 94.3757% in classifying 15 semantic classes.

Windformer: A novel 4D high-resolution system for multi-step wind speed vector forecasting based on temporal shifted window multi-head self-attention

Accurate wind speed forecasting (WSF) plays a pivotal role in anticipating the power output of wind farms. Nevertheless, the stochastic and variable nature of wind speed presents a significant challenge in achieving accurate WSF. Hence, this study proposes a novel 4D high-resolution system, Windformer, for wind speed vector forecasting (WSVF). Windformer combines the ability of convolutional neural networks (CNNs) for feature extraction and transformers based on attention mechanisms for information fusion. In the Windformer, the input and output layers are mainly composed of 3D CNNs. The input layer is employed for feature extraction and information compression, while the output layer is tasked with recovering the WSV field. The key components of the model consist of encoders and decoders built upon temporal shifted window multi-head self-attention. This architecture is capable of effectively integrating spatio-temporal information. Trained on 39 years of regional reanalysis data, Windformer obtains the most accurate forecast results compared to 5 WSF baseline models. Most importantly, within the next 6 hours, Windformer demonstrates higher accuracy compared to the High-Resolution Deterministic Forecast (HRES) from the European Centre for Medium-Range Weather Forecasts (ECMWF).

Sensitivity Matrix Update Method for Electrical Resistance Tomography Based on Error-Constrained Cross Fusion Residual Attention Network

Abstract Electrical resistance tomography (ERT) is an imaging technique of conductivity distribution. The image reconstruction algorithm based on the empty field sensitivity matrix is widely used because it provides an approximate solution to the complex ERT inverse problems. However, due to the soft field effect, the sensitivity matrix changes with the media distribution, leading to significant errors in image reconstruction. To address this issue, an error-constrained deep learning scheme for bubble flow, namely cross fusion residual attention network (CFRA-Net) is designed to update the sensitivity matrix during gas-liquid-solid fluidized bed operation. CFRA-Net employs a multi-head self-attention mechanism to enhance bubble features in boundary measurements, a multilevel cross fusion module to fuse empty field strength and boundary measurement information, and a novel serial-channel residual spatial attention block to boost the feature extraction capability of the model. A weighted loss function is designed to give more weight to the gas phase distribution region. Additionally, this paper establishes a dataset for gas-liquid-solid three-phase flow, considering background field conductivity variations. Validation and reliability of the proposed method are assessed through ablation, comparison, and noise experiments. The experimental results demonstrate that CFRA-Net achieves rapid updates of the sensitivity matrix, high imaging accuracy, and robust noise immunity.

Residual SwinV2 transformer coordinate attention network for image super resolution

Swin Transformers have been designed and used in various image super-resolution (SR) applications. One of the recent image restoration methods is RSTCANet, which combines Swin Transformer with Channel Attention. However, for some channels of images that may carry less useful information or noise, Channel Attention cannot automatically learn the insignificance of these channels. Instead, it tries to enhance their expression capability by adjusting the weights. It may lead to excessive focus on noise information while neglecting more essential features. In this paper, we propose a new image SR method, RSVTCANet, based on an extension of Swin2SR. Specifically, to effectively gather global information for the channel of images, we modify the Residual SwinV2 Transformer blocks in Swin2SR by introducing the coordinate attention for each two successive SwinV2 Transformer Layers (S2TL) and replacing Multi-head Self-Attention (MSA) with Efficient Multi-head Self-Attention version 2 (EMSAv2) to employ the resulting residual SwinV2 Transformer coordinate attention blocks (RSVTCABs) for feature extraction. Additionally, to improve the generalization of RSVTCANet during training, we apply an optimized RandAugment for data augmentation on the training dataset. Extensive experimental results show that RSVTCANet outperforms the recent image SR method regarding visual quality and measures such as PSNR and SSIM.

Modeling of joint extraction of entity relationships in clinical electronic medical records

The advancement of medical informatization necessitates extracting entities and their relationships from electronic medical records. Presently, research on electronic medical records predominantly concentrates on single-entity relationship extraction. However, clinical electronic medical records frequently exhibit overlapping complex entity relationships, thereby heightening the challenge of information extraction. To rectify the absence of a clinical medical relationship extraction dataset, this study utilizes electronic medical records from 584 patients in a hospital to create a compact clinical medical relationship extraction dataset. To address the pipelined relationship extraction model’s limitation in overlooking the one-to-many correlation problem between entities and relationships, this paper introduces a cascading relationship extraction model. This model integrates the MacBERT pre-training model, gated recurrent network, and multi-head self-attention mechanism to enhance the extraction of text features. Simultaneously, adversarial learning is incorporated to bolster the model’s robustness. In scenarios involving one-to-many relationships between entities, a two-phase task is employed. Initially, the main entity is predicted, followed by predicting the associated object and their correspondences. Employing this cascade-structured approach enables the model to flexibly manage intricate entity relationships, thereby enhancing extraction accuracy. Experimental results demonstrate the model’s efficiency, yielding F1-scores of 82.8%, 76.8%, and 88.2% for fulfilling relational extraction requirements and tasks on DuIE, CHIP-CDEE, and private datasets, respectively. These scores represent improvements over the benchmark model. The findings indicate the model’s applicability in practical domains, particularly in tasks such as biomedical information extraction.