Self-attention Mechanism Research Articles

Sgmsnet: self-guided multi-scale fusion network for remote sensing image scene classification

Remote sensing image scene classification (RSISC) is a key focus in the field of remote sensing image interpretation. Both CNN-Based and ViT-Based semantic feature extraction methods have been applied to RSISC. However, the challenges posed by complex scene images, such as high intraclass diversity and interclass similarity, remain significant obstacles to the feature extraction capabilities of network models. To address the aforementioned challenges, this study introduces the SGMSNet model. The first branch optimizes the key-value information in the self-attention mechanism, enabling ViT to effectively extract the global structural features of the target scene image while maintaining a lower network parameter cost. The second branch extracts multiple irregular local key features of the target scene image by constructing a lightweight pyramid network to supplement the feature loss of the first branch. Subsequently, a designed feature fusion module is employed to automatically adjust and merge the weights of global and local features for each scene image. The overall accuracy results on the UC Merced Land Use Dataset (UCM), the Aerial Image Dataset (AID), and the Northwestern Polytechnical University (NWPU)-RESISC45 Dataset were 99.17%, 97.43% and 94.87%, respectively. These results show that SGMSNet is suitable for the applications of with low network complexity requirements.

R2SCAT-LPR: Rotation-Robust Network with Self- and Cross-Attention Transformers for LiDAR-Based Place Recognition

LiDAR-based place recognition (LPR) is crucial for the navigation and localization of autonomous vehicles and mobile robots in large-scale outdoor environments and plays a critical role in loop closure detection for simultaneous localization and mapping (SLAM). Existing LPR methods, which utilize 2D bird’s-eye view (BEV) projections of 3D point clouds, achieve competitive performance in efficiency and recognition accuracy. However, these methods often struggle with capturing global contextual information and maintaining robustness to viewpoint variations. To address these challenges, we propose R2SCAT-LPR, a novel, transformer-based model that leverages self-attention and cross-attention mechanisms to extract rotation-robust place feature descriptors from BEV images. R2SCAT-LPR consists of three core modules: (1) R2MPFE, which employs weight-shared cascaded multi-head self-attention (MHSA) to extract multi-level spatial contextual patch features from both the original BEV image and its randomly rotated counterpart; (2) DSCA, which integrates dual-branch self-attention and multi-head cross-attention (MHCA) to capture intrinsic correspondences between multi-level patch features before and after rotation, enhancing the extraction of rotation-robust local features; and (3) a combined NetVLAD module, which aggregates patch features from both the original feature space and the rotated interaction space into a compact and viewpoint-robust global descriptor. Extensive experiments conducted on the KITTI and NCLT datasets validate the effectiveness of the proposed model, demonstrating its robustness to rotation variations and its generalization ability across diverse scenes and LiDAR sensors types. Furthermore, we evaluate the generalization performance and computational efficiency of R2SCAT-LPR on our self-constructed OffRoad-LPR dataset for off-road autonomous driving, verifying its deployability on resource-constrained platforms.

Multi-Label Auroral Image Classification Based on CNN and Transformer.

Auroral image classification has long been a focus of research in auroral physics. However, current methods for automatic auroral classification typically assume that only one type of aurora is present in an auroral image. This oversight neglects the complex transition states and coexistence of multiple types during the auroral evolution process, thus limiting the exploration of the intricate semantics of auroral images. To fully exploit the physical information embedded in auroral images, this paper proposes a multi-label auroral classification method, termed MLAC, which integrates convolutional neural network (CNN) and Transformer architectures. Firstly, we introduce a multi-scale feature fusion framework that enables the model to capture both fine-grained features and high-level information in auroral images, resulting in a more comprehensive representation of auroral features. Secondly, we propose a lightweight multi-head self-attention mechanism that captures long-range dependencies between pixels during the multiscale feature fusion process, which is crucial for distinguishing subtle differences between auroral types. Furthermore, we design a residual focused multilayer perceptron module that integrates large kernel depth-wise convolution with an improved multilayer perceptron. This integration enhances the model's ability to represent complex spatial structure, thus improving local feature extraction and global contextual understanding. The proposed method achieves a mean average precision (mAP) of 88.20% on the auroral observation data collected at the Yellow River Station from 2003 to 2008. This performance significantly surpasses that of the most advanced multi-label classification models while maintaining competitive computational efficiency. Moreover, our method also outperforms the state-of-the-art multi-label methods in both computational efficiency and classification accuracy on two publicly available multi-label image datasets: WIDER-Attribute and VOC2007. These results demonstrate that our method skillfully leverages the robust feature extraction capability of CNNs for local features and the superior global information processing capability of Transformer.

Extraction of fetal heartbeat locations in abdominal phonocardiograms using deep attention transformer.

Extraction of fetal heartbeat locations in abdominal phonocardiograms using deep attention transformer.

Lightweight obstacle detection for unmanned mining trucks in open-pit mines

This paper aims to solve the problem of the difficulty in balancing the model size and detection accuracy of the unmanned mining truck detection network in open-pit mines, as well as the problem that the existing model is not suitable for mining truck equipment. To address this problem, we proposed a lightweight vehicle detection algorithm model based on the improvement of YOLOv8. Through a series of innovative structural adjustments and optimization strategies, the model has achieved high accuracy and low complexity. This paper replaces the backbone network of YOLOv8s with the FasterNet_t0 (FN) network. This network has the advantages of simple structure and high lightweight, which effectively reduces the amount of calculation and parameters of the model. Then the feature extraction structure of the YOLOv8 neck is replaced with the BiFPN (Bi-directional Feature Pyramid Network) structure. By increasing cross-layer connections and removing nodes with low contribution to feature fusion, the fusion and utilization of features of different scales are optimized, the model performance is further improved, and the number of parameters and calculations are reduced. To make up for the possible loss of accuracy caused by lightweight improvements, this paper replaces the detection head with Dynamic Head. This design can introduce the self-attention mechanism from the three dimensions of scale, space, and task, significantly improving the detection accuracy of the model while avoiding the additional computational burden. In terms of loss function, this paper introduces a combination of SIoU loss and NWD (normalized Gaussian Wasserstein distance) loss. These two adjustments enable the model to cope with different scenarios more accurately, especially the detection effect of small target mining trucks is significantly improved. In addition, this paper also adopts the amplitude-based layer adaptive sparse pruning algorithm (LAMP) to further compress the model size while maintaining efficient detection performance. Through this pruning strategy, the model further reduces its dependence on computing resources while maintaining key performance. In the experimental part, a dataset of 3000 images was first constructed, and these images were preprocessed, including image enhancement, denoising, cropping, and scaling. The experimental environment was set up on the Autodl cloud server, using the PyTorch 2.5.1 framework and Python 3.10 environment. Through four sets of ablation experiments, we verified the specific impact of each improvement on the model performance. The experimental results show that the lightweight improvement strategy significantly improves the detection accuracy of the model, while greatly reducing the number of parameters and calculations of the model. Finally, we conducted a comprehensive comparative analysis of the improved YOLOv8s model with other popular algorithms and models. The results show that our model leads in detection accuracy with 76.9%, which is more than 10% higher than the performance of similar models. At the same time, compared with other models that achieve similar accuracy levels, our model is only about 20% of the size. These results fully prove that the improvement strategy we adopted is feasible and has obvious advantages in improving model efficiency.

GFSNet: Gaussian Fourier with sparse attention network for visual question answering

Visual question answering (VQA), a core task in multimodal learning, requires models to effectively integrate visual and natural language information to perform reasoning and semantic understanding in complex scenarios. However, self-attention mechanisms often struggle to capture multi-scale information and key region features within images comprehensively. Moreover, VQA involves multidimensional and deep reasoning about image content, particularly in scenarios involving spatial relationships and frequency-domain features. Existing methods face limitations in modeling multi-scale features and filtering irrelevant information effectively. This paper proposes an innovative Gaussian Fourier with Sparse Attention Network (GFSNet) to address these challenges. GFSNet leverages Fourier transforms to map image attention weights generated by the self-attention mechanism from the spatial domain to the frequency domain, enabling comprehensive modeling of multi-scale frequency information. This enhances the model’s adaptability to complex structures and its capacity for relational modeling. To further improve feature robustness, a Gaussian filter is introduced to suppress high-frequency noise in the frequency domain, preserving critical visual information. Additionally, a sparse attention mechanism dynamically selects optimized frequency-domain features, effectively reducing interference from redundant information while improving interpretability and computational efficiency. Without increasing parameter counts or computational complexity, GFSNet achieves efficient modeling of multi-scale visual information. Experimental results on benchmark VQA datasets (VQA v2, GQA, and CLEVR) demonstrate that GFSNet significantly enhances reasoning capabilities and cross-modal alignment performance, validating its superiority and effectiveness. The code is available at https://github.com/shenxiang-vqa/GFSNet.

Research on the Construction of Short-term Prediction Model of Wind Farm Output Power Based on Deep Learning Model

With the shift in the global energy structure, wind energy has become increasingly prominent in the energy market. However, wind power generation poses great challenges to grid dispatching due to its intermittency and randomness, so accurate wind power output prediction is of great significance to improve grid stability and reduce operating costs. Based on deep learning technology, this paper constructs a model for short-term power forecasting of wind farms, aiming to improve the accuracy of wind power forecasting. The model in this paper has been improved in data preprocessing, feature extraction and algorithm optimization. We used the actual operation data of a wind farm in a coastal area. The data covered three years of historical wind speed, wind direction and power output information, totaling more than 500,000 data records. The model adopts a long-term and short-term memory network architecture and combines self-attention mechanism to perform multi-step prediction. Finally, the average absolute error on the test set is reduced to 0.045, prediction accuracy 15% traditional statistical. This study shows that the application of deep learning in wind power forecasting has significant advantages and provides an important reference for the construction of smart grids in the future.

Neuron Perception Inspired EEG Emotion Recognition With Parallel Contrastive Learning.

Considerable interindividual variability exists in electroencephalogram (EEG) signals, resulting in challenges for subject-independent emotion recognition tasks. Current research in cross-subject EEG emotion recognition has been insufficient in uncovering the shared neural underpinnings of affective processing in the human brain. To address this issue, we propose the parallel contrastive multisource domain adaptation (PCMDA) model, inspired by the neural representation mechanism in the ventral visual cortex. Our model employs a neuron-perception-inspired contrastive learning architecture for EEG-based emotion recognition in subject-independent scenarios. A two-stage alignment methodology is employed for the purpose of aligning numerous source domains with the target domain. This approach integrates a parallel contrastive loss (PCL) which simulates the self-supervised learning mechanism inherent in the neural representation of the human brain. Furthermore, a self-attention mechanism is integrated to extract emotion weights for each frequency band. Extensive experiments were conducted on three publicly available EEG emotion datasets, SJTU emotion EEG dataset (SEED), database for emotion analysis using physiological signals (DEAP), and finer-grained affective computing EEG dataset (FACED), to evaluate our proposed method. The results demonstrate that the PCMDA effectively utilizes the unique EEG features and frequency band information of each subject, leading to improved generalization across different subjects in comparison to other methods.

Energy trading optimisation of microgrids considering degradation: a self-attention-based deep reinforcement learning method

The trading environment in microgrids is complicated by the uncertainty of renewable energy generation and the degradation of Energy Storage Systems (ESS), which significantly reduces the stability and accuracy of energy trading decisions, leading to the loss of system benefits. A novel Energy Trading Model considering ESS degradation (ETMED) is established to maximise system benefits, and then transformed into a Markov game problem. Self-Attention based Double Deep Q Network (SA-DDQN) is proposed to solve it. In SA-DDQN, the self-attention mechanism weights the state information, enhancing the algorithm's ability to capture the internal relationship between state data. Furthermore, the duelling architecture divides the value function into the state value function and the action advantage function, enabling the algorithm to better adapt to the complex trading environment and make accurate trading decisions, thus increasing the system benefits. Compared to the baseline algorithms, simulation results demonstrate that SA-DDQN obtains higher system benefits with better convergence.

Privacy-Preserving Text Classification on Deep Neural Network

With the explosive growth of Internet information, the classification of massive Internet data plays a very important role in real life. Text classification has been widely used in spam text recognition, intention recognition, text matching, named entity recognition, and other fields. At present, many enterprises provide APIs for text classification for users. Users can upload their data to the cloud server deployed by service providers for analysis, and return the final classification results. However, there is a risk of user data and model leakage in this process. To solve this problem, we propose a privacy-preserving text classification scheme using CKKS fully homomorphic encryption scheme and self-attention mechanism model in the multi-party security computing scenario. Our scheme ensures that user can achieve efficient encrypted data analysis under the premise of their data security, and user must be authorized by the service provider to use the model. Finally, compared with the experimental results of the previous research on privacy text classification under fully homomorphic encryption, the implementation improves the accuracy by 7.97% at most and speed-ups 282.4 times for inference at most, and we ensure the security of the protocol participants.

Enhancing object pose estimation for RGB images in cluttered scenes

Estimating the 6D pose of objects is crucial for robots to interact with the environment. 6D Object pose estimation from RGB images in a cluttered scene and heavy occlusions is a critical issue. Most existing methods use two stages to estimate object pose: First, extract the object features, and then use the PnP/RANSAC method to estimate object pose. However, most of these techniques merely localize a group of key-points by regressing their coordinates, which are vulnerable to occlusion and have poor performance for multi-object pose estimation. These methods cannot directly regress the 6D pose estimation from a loss during training. In this paper, we propose a framework based on convolutional neural network (CNN) and self-attention mechanism as an end-to-end method for single and multi-object 6D pose estimation using RGB images with low computational cost. Our method utilizes feature fusion to extract local features and combines multi-head self-attention (MHSA) with iterative refinement to improve pose estimation performance. Furthermore, our method can be scaled according to computational resources. Our experiments illustrate that our method performs in benchmark datasets the Linemod and Occlusion Linemod and achieves 97.45% and 84.84% in terms of the ADD(-S) metric in both datasets, respectively.

The Heterogeneous Network Community Detection Model Based on Self-Attention

With the advancement of representation learning, graph representation learning has gained significant attention in the field of community detection for heterogeneous networks. A prominent approach in this domain involves the use of meta-paths to capture higher-order relationships between nodes, particularly when bidirectional or reciprocal relationships exist. However, defining effective meta-paths often requires substantial domain expertise. Moreover, these methods typically depend on additional clustering algorithms, which can limit their practical applicability. To address these challenges, context paths have been introduced as an alternative to meta-paths. When combined with a self-attention mechanism, models can dynamically assess the relative importance of different context paths. By leveraging the inherent symmetry within context paths, models enhance their ability to capture balanced relationships between nodes, thereby improving their representation of complex interactions. Building on this idea, we propose BP-GCN, a self-attention-based model for heterogeneous community detection. BP-GCN autonomously identifies node relationships within symmetric context paths, significantly improving community detection accuracy. Furthermore, the model integrates the Bernoulli–Poisson framework to establish an end-to-end detection system that eliminates the need for auxiliary clustering algorithms. Extensive experiments on multiple real-world datasets demonstrate that BP-GCN consistently outperforms existing benchmark methods.

An improved transformer based traffic flow prediction model

Traffic flow prediction is a key challenge in intelligent transportation, and the ability to accurately forecast future traffic flow directly affects the efficiency of urban transportation systems. However, existing deep learning-based prediction models suffer from the following issues: First, CNN- or RNN-based models are limited by their architecture and unsuitable for modeling long-term sequences. Second, most Transformer-based methods focus solely on the traffic flow data itself during embedding, neglecting the implicit information behind the traffic data. This implicit information includes behavioral trends, community and surrounding traffic patterns, urban weather, semantic information, and temporal periodicity. Third, methods using the original multi-head self-attention mechanism calculate attention scores point by point in the temporal dimension without utilizing contextual information, which to some extent leads to less accurate attention computation. Fourth, existing methods struggle to capture long and short-range spatial dependencies simultaneously. To address these four issues, we propose an IEEAFormer technique (Implicit-information Embedding and Enhanced Spatial-Temporal Multi-Head Attention Transformer). First, it adopts a Transformer architecture and incorporates an embedding layer to capture implicit information in the input. Secondly, the method replaces the traditional multi-head self-attention with time-environment-aware self-attention in the temporal dimension, enabling each node to perceive the contextual environment. Additionally, the technique uses two unique graph mask matrices in the spatial dimension. It employs a novel parallel spatial self-attention architecture to capture both long-range and short-range dependencies in the data simultaneously. The results verified on four real-world traffic datasets show that the proposed IEEAFormer outperforms most existing models regarding prediction performance.

End-to-End Vector Simplification for Building Contours via a Sequence Generation Model

Simplifying building contours involves reducing data volume while preserving the continuity, accuracy, and essential characteristics of building shapes. This presents significant challenges for sequence representation and generation. Traditional methods often rely on complex rule design, feature engineering, and iterative optimization. To overcome these limitations, this study proposes a Transformer-based Polygon Simplification Model (TPSM) for the end-to-end vector simplification of building contours. TPSM processes ordered vertex coordinate sequences of building contours, leveraging the inherent sequence modeling capabilities of the Transformer architecture to directly generate simplified coordinate sequences. To enhance spatial understanding, positional encoding is embedded within the multihead self-attention mechanism, allowing the TPSM to effectively capture relative vertex positions. Additionally, a self-supervised reconstruction mechanism is introduced, where random perturbations are applied to input sequences, and the model learns to reconstruct the original contours. This mechanism enables TPSM to better understand underlying geometric relationships and implicit simplification rules. Experiments were conducted using a 1:10,000 building dataset from Shenzhen, China, targeting a simplification scale of 1:25,000. The results demonstrate that TPSM outperforms five established simplification algorithms in controlling changes to building area, orientation, and shape fidelity, achieving an average intersection over union (IoU) of 0.901 and a complexity-aware IoU (C-IoU) of 0.735.

Multi-Channel Speech Enhancement Using Labelled Random Finite Sets and a Neural Beamformer in Cocktail Party Scenario

In this research, a multi-channel target speech enhancement scheme is proposed that is based on deep learning (DL) architecture and assisted by multi-source tracking using a labeled random finite set (RFS) framework. A neural network based on minimum variance distortionless response (MVDR) beamformer is considered as the beamformer of choice, where a residual dense convolutional graph-U-Net is applied in a generative adversarial network (GAN) setting to model the beamformer for target speech enhancement under reverberant conditions involving multiple moving speech sources. The input dataset for this neural architecture is constructed by applying multi-source tracking using multi-sensor generalized labeled multi-Bernoulli (MS-GLMB) filtering, which belongs to the labeled RFS framework, to obtain estimations of the sources’ positions and the associated labels (corresponding to each source) at each time frame with high accuracy under the effect of undesirable factors like reverberation and background noise. The tracked sources’ positions and associated labels help to correctly discriminate the target source from the interferers across all time frames and generate time–frequency (T-F) masks corresponding to the target source from the output of a time-varying, minimum variance distortionless response (MVDR) beamformer. These T-F masks constitute the target label set used to train the proposed deep neural architecture to perform target speech enhancement. The exploitation of MS-GLMB filtering and a time-varying MVDR beamformer help in providing the spatial information of the sources, in addition to the spectral information, within the neural speech enhancement framework during the training phase. Moreover, the application of the GAN framework takes advantage of adversarial optimization as an alternative to maximum likelihood (ML)-based frameworks, which further boosts the performance of target speech enhancement under reverberant conditions. The computer simulations demonstrate that the proposed approach leads to better target speech enhancement performance compared with existing state-of-the-art DL-based methodologies which do not incorporate the labeled RFS-based approach, something which is evident from the 75% ESTOI and PESQ of 2.70 achieved by the proposed approach as compared with the 46.74% ESTOI and PESQ of 1.84 achieved by Mask-MVDR with self-attention mechanism at a reverberation time (RT60) of 550 ms.

Adaptive Transformer-Based Deep Learning Framework for Continuous Sign Language Recognition and Translation

Sign language recognition and translation remain pivotal for facilitating communication among the deaf and hearing communities. However, end-to-end sign language translation (SLT) faces major challenges, including weak temporal correspondence between sign language (SL) video frames and gloss annotations and the complexity of sequence alignment between long SL videos and natural language sentences. In this paper, we propose an Adaptive Transformer (ADTR)-based deep learning framework that enhances SL video processing for robust and efficient SLT. The proposed model incorporates three novel modules: Adaptive Masking (AM), Local Clip Self-Attention (LCSA), and Adaptive Fusion (AF) to optimize feature representation. The AM module dynamically removes redundant video frame representations, improving temporal alignment, while the LCSA module learns hierarchical representations at both local clip and full-video levels using a refined self-attention mechanism. Additionally, the AF module fuses multi-scale temporal and spatial features to enhance model robustness. Unlike conventional SLT models, our framework eliminates the reliance on gloss annotations, enabling direct translation from SL video sequences to spoken language text. The proposed method was evaluated using the ArabSign dataset, demonstrating state-of-the-art performance in translation accuracy, processing efficiency, and real-time applicability. The achieved results confirm that ADTR is a highly effective and scalable deep learning solution for continuous sign language recognition, positioning it as a promising AI-driven approach for real-world assistive applications.

Application of CNN-BiGRU-MHSA: A Self-Attention Mechanism Based Detection Method in Electronic Data Forensics

Application of CNN-BiGRU-MHSA: A Self-Attention Mechanism Based Detection Method in Electronic Data Forensics

Application of deep learning in biomechanical image recognition: Based on transformer architecture

Biomechanical image recognition has important applications in clinical diagnosis and biomedical engineering, but traditional convolutional neural network (CNN) has limitations in capturing global features. In this paper, a biomechanical image recognition method based on Vision Transformer (ViT) is proposed to improve the classification performance of complex images. Biomechanical image dataset containing five types of data is constructed, and ViT input features are represented by standardization, data enhancement and Patch segmentation. Accuracy, precision, recall, F1 score and confusion matrix are used to evaluate the performance, and compared with ResNet-50 and DenseNet-121. The experimental results show that the accuracy of ViT model is 92.3%, and it performs best in the categories of “normal bones” and “soft tissue lesions”, and other indicators are better than the traditional CNN model. ViT realizes global feature modeling through self-attention mechanism, which significantly improves the recognition accuracy and robustness, provides efficient and accurate technical support for clinical diagnosis, disease screening and surgical planning, and shows its application potential in the field of biomechanical image recognition.

MySTOCKS: Multi-Modal Yield eSTimation System of in-prOmotion Commercial Key-ProductS

In recent years, Out-of-Stock (OOS) occurrences have posed a persistent challenge for both retailers and manufacturers. In the context of grocery retail, an OOS event represents a situation where customers are unable to locate a specific product when attempting to make a purchase. This study analyzes the issue from the manufacturer’s perspective. The proposed system, named the “Multi-modal yield eSTimation System of in-prOmotion Commercial Key-ProductS” (MySTOCKS) platform, is a sophisticated multi-modal yield estimation system designed to optimize inventory forecasting for the agrifood and large-scale retail sectors, particularly during promotional periods. MySTOCKS addresses the complexities of inventory management in settings where Out-of-Stock (OOS) and Surplus-of-Stock (SOS) situations frequently arise, offering predictive insights into final stock levels across defined forecasting intervals to support sustainable resource management. Unlike traditional approaches, MySTOCKS leverages an advanced deep learning framework that incorporates transformer models with self-attention mechanisms and domain adaptation capabilities, enabling accurate temporal and spatial modeling tailored to the dynamic requirements of the agrifood supply chain. The system includes two distinct forecasting modules: TR1, designed for standard stock-level estimation, and TR2, which focuses on elevated demand periods during promotions. Additionally, MySTOCKS integrates Elastic Weight Consolidation (EWC) to mitigate the effects of catastrophic forgetting, thus enhancing predictive accuracy amidst changing data patterns. Preliminary results indicate high system performance, with test accuracy, sensitivity, and specificity rates approximating 93.8%. This paper provides an in-depth examination of the MySTOCKS platform’s modular structure, data-processing workflow, and its broader implications for sustainable and economically efficient inventory management within agrifood and large-scale retail environments.

Fault diagnosis in electric motors using multi-mode time series and ensemble transformers network.

Induction motors are essential in industrial production, and their fault diagnosis is vital for ensuring continuous and efficient equipment operation. Minimizing downtime losses and optimizing maintenance costs are key to maintaining smooth production and enhancing economic efficiency. This paper presents a novel diagnostic approach for diverse motor faults, integrating time series analysis, Transformer-based networks, and multi-modal data fusion. Firstly, multiple signals such as three-phase current, vibration, device sound, and ambient sound are collected to form a multi-modal dataset. Subsequently, a Transformer network for single time series classification is developed, and multiple instances are concatenated in parallel to create an ensemble Transformer network. The self-attention mechanism is then utilized to dynamically integrate features from different modal data for accurate motor fault identification. During network training, the chaotic WOA optimizes the ensemble Transformer network's hyper-parameters. Finally, the proposed method is trained and tested on a motor measurement multi-modal dataset. Experimental results show that it performs outstandingly on multi-modal datasets, attaining a high diagnostic accuracy of 99.10%. Compared with single-mode data and state-of-the-art methods, it demonstrates superior diagnostic accuracy and reliability.