Transformer Model Research Articles

Paying attention to uncertainty: A stochastic multimodal transformers for post-traumatic stress disorder detection using video

Background and Objectives:Post-traumatic stress disorder is a debilitating psychological condition that can manifest following exposure to traumatic events. It affects individuals from diverse backgrounds and is associated with various symptoms, including intrusive thoughts, nightmares, hyperarousal, and avoidance behaviors. Methods:To address this challenge this study proposes a decision support system powered by a novel multimodal deep learning approach, based on a stochastic Transformer and video data. This Transformer has the ability to take advantage of its stochastic activation function and layers that allow it to learn sparse representations of the inputs. The method leverages a combination of low-level features extracted using three modalities, including Mel-frequency cepstral coefficients extracted from audio recordings, Facial Action Units captured from facial expressions, and textual data obtained from the audio transcription. By considering these modalities, our proposed model captures a comprehensive range of information related to post-traumatic stress disorder symptoms, including vocal cues, facial expressions, and linguistic content. Results:The deep learning model was trained and evaluated on the eDAIC dataset, which consists of clinical interviews with individuals with and without post-traumatic disorder. The model achieved state-of-the-art results, demonstrating its effectiveness in accurately detecting PTSD, showing an impressive Root Mean Square Error of 1.98, and a Concordance Correlation Coefficient of 0.722, signifying the model’s superior performance compared to existing approaches. Conclusion:This work introduces a new method for post-traumatic stress disorder detection from videos by utilizing a multimodal stochastic Transformer model. The model makes use of a variety of modalities, such as text, audio, and visual data, to gather comprehensive and varied information in order to make the detection.

SparseMorph: A weakly-supervised lightweight sparse transformer for mono- and multi-modal deformable image registration

PurposeDeformable image registration (DIR) is crucial for improving the precision of clinical diagnosis. Recent Transformer-based DIR methods have shown promising performance by capturing long-range dependencies. Nevertheless, these methods still grapple with high computational complexity. This work aims to enhance the performance of DIR in both computational efficiency and registration accuracy. MethodsWe proposed a weakly-supervised lightweight Transformer model, named SparseMorph. To reduce computational complexity without compromising the representative feature capture ability, we designed a sparse multi-head self-attention (SMHA) mechanism. To accumulate representative features while preserving high computational efficiency, we constructed a multi-branch multi-layer perception (MMLP) module. Additionally, we developed an anatomically-constrained weakly-supervised strategy to guide the alignment of regions-of-interest in mono- and multi-modal images. ResultsWe assessed SparseMorph in terms of registration accuracy and computational complexity.Within the mono-modal brain datasets IXI and OASIS, our SparseMorph outperforms the state-of-the-art method TransMatch with improvements of 3.2 % and 2.9 % in DSC scores for MRI-to-CT registration tasks, respectively. Moreover, in the multi-modal cardiac dataset MMWHS, our SparseMorph shows DSC score improvements of 9.7 % and 11.4 % compared to TransMatch in MRI-to-CT and CT-to-MRI registration tasks, respectively. Notably, SparseMorph attains these performance advantages while utilizing 33.33 % of the parameters of TransMatch. ConclusionsThe proposed weakly-supervised deformable image registration model, SparseMorph, demonstrates efficiency in both mono- and multi-modal registration tasks, exhibiting superior performance compared to state-of-the-art algorithms, and establishing an effective DIR method for clinical applications.

Artificial intelligence based real time colorectal cancer screening study: Polyp segmentation and classification using multi-house database

PurposeThis work explores the potential of artificial intelligence (AI) for real-time colorectal cancer (CRC) screening. We utilized a multi-house dataset to achieve a three-step approach (i) automated polyp segmentation, (ii) analysis of machine learning (ML) for classification and (iii) to compare deep learning (DL) exploration for automated classification. MethodsTo achieve our objectives, we leveraged the CRPU-Net architecture for automated polyp segmentation within the multi-house colonoscopy images. Subsequently, a ML framework was employed for binary classification of the segmented regions. Feature extraction was performed using SIFT and ORB descriptors, followed by the feature fusion and dimensionality reduction through Principal Component Analysis (PCA) to create an optimal feature representation for classification. We further explored the potential of DL by investigating the use of SOTA and vision transformer (ViT) model. Additionally, we considered the development of a hybrid model (CRP-ViT) by potentially combining the ViT model with DL architectures. ResultsCRPU-Net achieved outstanding segmentation performance of 96.56% accuracy, 95.40% IoU, and 97.39% Dice coefficient. Additionally, the CRP-ViT model demonstrated promising results in binary classification of 96.59% accuracy, 96.38% sensitivity, 96.80% specificity, 96.82% precision, and 96.36% NPV. ConclusionThis study implemented CRPU-Net and CRP-ViT, AI systems designed for real-time polyp segmentation, and potential precancerous polyp classification. This comprehensive approach offers the potential to expedite colonoscopies, minimize unnecessary biopsies, improve patient care, and optimize healthcare resource allocation.

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals.

This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train subject-specific models using data from a single participant and multi-patient models exploiting data from multiple participants. The subject-specific models using only low-density 8×8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. The proposed SwinTW decoder enables future speech neuropros-theses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests that such a model can be applied to new patients that do not have paired acoustic and neural data, providing an advance in neuroprostheses for people with speech disability, where acoustic-neural training data is not feasible.

A Simplified Query-Only Attention for Encoder-Based Transformer Models

A Simplified Query-Only Attention for Encoder-Based Transformer Models

Representation Learning for Sequential Volumetric Design Tasks

Abstract Volumetric design, also called massing design, is the first and critical step in professional building design which is sequential in nature. As the volumetric design process is complex, the underlying sequential design process en- codes valuable information for designers. Many efforts have been made to automatically generate reasonable volumetric designs, but the quality of the generated design solutions varies, and evaluating a design solution requires either a prohibitively comprehensive set of metrics or expensive human expertise. While previous approaches focused on learning only the final design instead of sequential design tasks, we propose to encode the design knowledge from a collection of expert or high-performing design sequences and extract useful representations using transformer-based models. Later we propose to utilize the learned representations for crucial downstream applications such as design preference evaluation and procedural design generation. We develop the preference model by estimating the density of the learned representations whereas we train an autoregressive transformer model for sequential design generation. We demonstrate our ideas by leveraging a novel dataset of thousands of sequential volumetric designs. Our preference model can compare two arbitrarily given design sequences and is almost 90% accurate in evaluation against random design sequences. Our autoregressive model is also capable of autocompleting a volumetric design sequence from a partial design sequence.

BaseNet: A Transformer-Based Toolkit for Nanopore Sequencing Signal Decoding

Nanopore sequencing provides a rapid, convenient and high-throughput solution for nucleic acid sequencing. Accurate basecalling in nanopore sequencing is crucial for downstream analysis. Traditional approaches such as Hidden Markov Models (HMM), Recurrent Neural Networks (RNN), and Convolutional Neural Networks (CNN) have improved basecalling accuracy but there is a continuous need for higher accuracy and reliability. In this study, we introduce BaseNet (https://github.com/liqingwen98/BaseNet), an open-source toolkit that utilizes transformer models for advanced signal decoding in nanopore sequencing. BaseNet incorporates both autoregressive and non-autoregressive transformer-based decoding mechanisms, offering state-of-the-art algorithms freely accessible for future improvement. Our research indicates that cross-attention weights effectively map the relationship between current signals and base sequences, joint loss training through adding a pair of forward and reverse decoder facilitate model converge, and large-scale pre-trained models achieve superior decoding accuracy. This study helps to advance the field of nanopore sequencing signal decoding, contributes to technological advancements, and provides novel concepts and tools for researchers and practitioners.

DeConformer-SENet: An efficient deformable conformer speech enhancement network

The Conformer model has demonstrated superior performance in speech enhancement by combining the long-range relationship modeling capability of self-attention with the local information processing ability of convolutional neural networks (CNNs). However, existing Conformer-based speech enhancement models struggle to balance performance and model complexity. In this work, we propose, DeConformer-SENet, an end-to-end time-domain deformable Conformer speech enhancement model, with modifications to both the self-attention and CNN components. Firstly, we introduce the time-frequency-channel self-attention (TFC-SA) module, which compresses information from each dimension of the input features into a one-dimensional vector. By calculating the energy distribution, this module models long-range relationships across three dimensions, reducing computational complexity while maintaining performance. Additionally, we replace standard convolutions with deformable convolutions, aiming to expand the receptive field of the CNN and accurately model local features. We validate our proposed DeConformer-SENet on the WSJ0-SI84 + DNS Challenge dataset. Experimental results demonstrate that DeConformer-SENet outperforms existing Conformer and Transformer models in terms of ESTOI and PESQ metrics, while also being more computationally efficient. Furthermore, ablation studies confirm that DeConformer-SENet improvements enhance the performance of conventional Conformer and reduce model complexity without compromising the overall effectiveness.

Fractal Self-Similarity in Semantic Convergence: Gradient of Embedding Similarity across Transformer Layers

This paper presents a mathematical analysis of semantic convergence in transformer-based language models, drawing inspiration from the concept of fractal self-similarity. We introduce and prove a novel theorem characterizing the gradient of embedding similarity across layers. Specifically, we establish that there exists a monotonically increasing function that provides a lower bound on the rate at which the average cosine similarity between token embeddings at consecutive layers and the final layer increases. This establishes a fundamental property: semantic alignment of token representations consistently increases through the network, exhibiting a pattern of progressive refinement, analogous to fractal self-similarity. The key challenge addressed is the quantification and generalization of semantic convergence across diverse model architectures and input contexts. To validate our findings, we conduct experiments on BERT and DistilBERT models, analyzing embedding similarities for diverse input types. While our experiments are limited to these models, we empirically demonstrate consistent semantic convergence within these architectures. Quantitatively, we find that the average rates of semantic convergence are approximately 0.0826 for BERT and 0.1855 for DistilBERT. We observe that the rate of convergence varies based on token frequency and model depth, with rare words showing slightly higher similarities (differences of approximately 0.0167 for BERT and 0.0120 for DistilBERT). This work advances our understanding of transformer models’ internal mechanisms and provides a mathematical framework for comparing and optimizing model architectures.

Classification Model of Grassland Desertification Based on Deep Learning

Grasslands are one of the most important ecosystems on earth, and the impact of grassland desertification on the earth’s environment and ecosystem cannot be ignored. Accurately distinguishing grassland desertification types has important application value. The appropriate grazing strategies can be implemented based on these distinctions. Grassland conservation measures can be tailored accordingly. This contributes to further protecting and restoring grassland vegetation. This project takes color images labeled with the desertification types of grasslands as the research object, uses the currently popular deep learning model as the classification tool, and then establishes a color image-based grassland desertification classification model based on the feature extraction network, based on the Vision Transformer model, by comparing the various deep learning image classification models. The experimental results show that, despite the complex structure and large number of parameters of the grassland desertification classification model obtained in this project, the test accuracy rate reaches 88.72% and the training loss is only 0.0319. Compared with the popular classification models such as VGG16, ResNet50, ResNet101, DenseNet101, DenseNet169, and DenseNet201, and so on, the Vision Transformer demonstrates clear advantages in classification accuracy, fitting ability, and generalization capacity. By integrating with deep learning technology, the model can be applied to grassland management and ecological restoration. Mobile devices can be used to conveniently capture image data, and information can be processed quickly. This provides efficient tools for grazing managers, environmental scientists, and conservation organizations. These tools assist in quickly assessing the extent of grassland desertification, optimizing grassland management and conservation decisions. Furthermore, strong technical support is offered for the ecological restoration and sustainable management of desertification grasslands.

A Hybrid Machine Learning Approach: Analyzing Energy Potential and Designing Solar Fault Detection for an AIoT-Based Solar–Hydrogen System in a University Setting

This research aims to optimize the solar–hydrogen energy system at Kangwon National University’s Samcheok campus by leveraging the integration of artificial intelligence (AI), the Internet of Things (IoT), and machine learning. The primary objective is to enhance the efficiency and reliability of the renewable energy system through predictive modeling and advanced fault detection techniques. Key elements of the methodology include data collection from solar energy production and fault detection systems, energy potential analysis using Transformer models, and fault identification in solar panels using CNN and ResNet-50 architectures. The Transformer model was evaluated using metrics such as Mean Absolute Error (MAE), Mean Squared Error (MSE), and an additional variation of MAE (MAE2). Known for its ability to detect intricate time series patterns, the Transformer model exhibited solid predictive performance, with the MAE and MAE2 results reflecting consistent average errors, while the MSE pointed to areas with larger deviations requiring improvement. In fault detection, the ResNet-50 model outperformed VGG-16, achieving 85% accuracy and a 42% loss, as opposed to VGG-16’s 80% accuracy and 78% loss. This indicates that ResNet-50 is more adept at detecting and classifying complex faults in solar panels, although further refinement is needed to reduce error rates. This study demonstrates the potential for AI and IoT integration in renewable energy systems, particularly within academic institutions, to improve energy management and system reliability. Results suggest that the ResNet-50 model enhances fault detection accuracy, while the Transformer model provides valuable insights for strategic energy output forecasting. Future research could focus on incorporating real-time environmental data to improve prediction accuracy and developing automated AIoT-based monitoring systems to reduce the need for human intervention. This study provides critical insights into advancing the efficiency and sustainability of solar–hydrogen systems, supporting the growth of AI-driven renewable energy solutions in university settings.

Multiagent trajectory decision‐making for complex systems based on transformer large model structure

AbstractThe article explores the application of transformer‐based large models for trajectory decision‐making in complex decision systems, focusing on multiagent nodes in a topological structure. By leveraging the capabilities of large models, the proposed approach enhances decision‐making in model‐free control environments characterized by numerous agents, diverse internal tasks, and unknown conditions. The proposed framework integrates advanced transformer models with a sophisticated decision‐making framework to enhance the efficiency, safety, and reliability of multiagent operations in dynamic environments. The results demonstrate the superior performance of the transformer model, achieving a trajectory decision‐making accuracy of 97.8%, significantly outperforming other models such as long short‐term memory (LSTM), gate recurrent unit (GRU), and traditional approaches. The visualizations highlight the close alignment between the true and predicted outcomes of complex decision‐making systems, showcasing the model's ability to capture complex dependencies and interactions within the multiagent environment. High task completion rates and low collision rates further underscore the effectiveness of the decision‐making framework in optimizing agent coordination and ensuring safe operations.

Graph-enabled spatio-temporal transformer for ionospheric prediction

The ionosphere, crucial for satellite navigation and radio communication, is highly sensitive to solar and geomagnetic activities. Traditional ionospheric prediction models often struggle to capture its complexities. This paper introduces a Graph-Enabled Spatio-temporal transformer (GEST) model, designed to predict the Total electron content (TEC) in the ionosphere with higher accuracy. The GEST model innovatively integrates six dynamic space weather parameters, including the Kp index, and refines grid distances and weights in the Global ionospheric maps (GIM). Leveraging the efficiency of the Transformer architecture in processing spatio-temporal data, the GEST model adapts to varying ionospheric conditions and provides superior precision. Our evaluations, based on data from the Center for orbit determination in Europe (CODE), demonstrate that the GEST model significantly outperforms traditional methods and other intelligent algorithms such as conv-LSTM. It shows marked improvements in Mean absolute error (MAE), Root mean square error (RMSE), Pearson correlation coefficient (CC), and Mean absolute percentage error (MAPE), particularly during periods of intense solar and geomagnetic activity. These advancements highlight the GEST model’s substantial contributions to the field of ionospheric prediction technology, offering a robust tool for enhancing the reliability of satellite-based systems.

Predicting pedestrian-vehicle interaction severity at unsignalized intersections

Objectives This study aims to develop and validate a novel deep-learning model that predicts the severity of pedestrian-vehicle interactions at unsignalized intersections, distinctively integrating Transformer-based models with Multilayer Perceptrons (MLP). This approach leverages advanced feature analysis capabilities, offering a more direct and interpretable method than traditional models. Methods High-resolution optical cameras recorded detailed pedestrian and vehicle movements at study sites, with data processed to extract trajectories and convert them into real-world coordinates via precise georeferencing. Trained observers categorized interactions into safe passage, critical event, and conflict based on movement patterns, speeds, and accelerations. Fleiss Kappa statistic measured inter-rater agreement to ensure evaluator consistency. This study introduces a novel deep-learning model combining Transformer-based time series data capabilities with the classification strengths of a Multilayer Perceptron (MLP). Unlike traditional models, this approach focuses on feature analysis for greater interpretability. The model, trained on dynamic input variables from trajectory data, employs attention mechanisms to evaluate the significance of each input variable, offering deeper insights into factors influencing interaction severity. Results The model demonstrated high performance across different severity categories: safe interactions achieved a precision of 0.78, recall of 0.91, and F1-score of 0.84. In more severe categories like critical events and conflicts, precision and recall were even higher. Overall accuracy stood at 0.87, with both macro and weighted averages for precision, recall, and F1-score also at 0.87. The variable importance analysis, using attention scores from the proposed transformer model, identified ‘Vehicle Speed’ as the most significant input variable positively influencing severity. Conversely, ‘Approaching Angle’ and ‘Vehicle Distance from Conflict Point’ negatively impacted severity. Other significant factors included ‘Type of Vehicle’, ‘Pedestrian Speed’, and ‘Pedestrian Yaw Rate’, highlighting the complex interplay of behavioral and environmental factors in pedestrian-vehicle interactions. Conclusions This study introduces a deep-learning model that effectively predicts the severity of pedestrian-vehicle interactions at crosswalks, utilizing a Transformer-MLP hybrid architecture with high precision and recall across severity categories. Key factors influencing severity were identified, paving the way for further enhancements in real-time analysis and broader safety assessments in urban settings.

Advancing semantic segmentation of two-dimensional materials using a semantic-adaptive transformer model

Accurate detection and characterization of two-dimensional (2D) materials are essential for their effective utilization in various applications. Traditional techniques, such as chemical vapor deposition, often produce materials with high defect density, while mechanical exfoliation is hindered by its labor-intensive and time-consuming nature. In this Letter, we propose a semantic-adaptive transformer model, termed Semptive, designed specifically for the precise detection of monolayer MoS2. Our approach integrates a semantic adaptation module with a multi-head self-attention mechanism, incorporating deep supervision and leveraging prior knowledge to enhance model performance. The model was trained on a dataset obtained through mechanical exfoliation and validated using photoluminescence spectroscopy. The experimental results reveal that Semptive significantly enhances segmentation performance compared to conventional models, achieving higher Intersection over Union and Dice scores while reducing computational demands. This method represents a notable advancement in the efficient and precise identification of 2D materials, providing substantial improvements for material characterization and device fabrication processes.

An illumination-guided dual attention vision transformer for low-light image enhancement

Existing Retinex-based low-light image enhancement methods often overlook corruptions hidden in darkness or pattern collapse caused by the lit-up process. Recent deep learning approaches suggest the use of U-shaped networks with Vision in Transformer (VIT) to address these issues. However, most VIT-based methods focus on channel modeling to reduce expensive computational costs, but in which the restored images suffer from spatial illumination inconsistencies, artifacts, and blurriness. To end for this, we propose a novel one-stage Retinex-based Illumination-Guided Dual transformer model (IGDFormer) to lit up low-light images. The model consists of an estimator and a restorer. The estimator generates a light-up feature map and a lit-up image through pure Convolutional Neural Networks (CNNs). The restorer denoises the lit-up image with a U-shaped network equipped with an Illumination-Guided Dual Attention Block (IGDAB). Specifically, IGDAB consists of cascaded channel attention and window attention that achieves cross-channel/spatial modeling. Channel attention alleviates inductive bias through the CNN-Transformer collaborative layer, and window attention introduces spatial domains knowledge by partitioning and shifting. In addition, the light-up features act as values guide the interaction modeling of non-local illumination intensities in both the channel and spatial domains. Extensive experiments were conducted on 5 low-light image enhancement benchmarks and 1 dark object detection benchmark, which demonstrate that the efficacy of our IGDFormer and its superiority in restoring spatial details compared to other state-of-the-art methods. The code is available at https://github.com/YanJieWen/IGDFormer-light-up-dark.

PathMethy: an interpretable AI framework for cancer origin tracing based on DNA methylation.

Despite advanced diagnostics, 3%-5% of cases remain classified as cancer of unknown primary (CUP). DNA methylation, an important epigenetic feature, is essential for determining the origin of metastatic tumors. We presented PathMethy, a novel Transformer model integrated with functional categories and crosstalk of pathways, to accurately trace the origin of tumors in CUP samples based on DNA methylation. PathMethy outperformed seven competing methods in F1-score across nine cancer datasets and predicted accurately the molecular subtypes within nine primary tumor types. It not only excelled at tracing the origins of both primary and metastatic tumors but also demonstrated a high degree of agreement with previously diagnosed sites in cases of CUP. PathMethy provided biological insights by highlighting key pathways, functional categories, and their interactions. Using functional categories of pathways, we gained a global understanding of biological processes. For broader access, a user-friendly web server for researchers and clinicians is available at https://cup.pathmethy.com.

Novel deep‐learning model for chemical process fault detection based on DCW transformer

AbstractIn this study, a double‐channel convolutional neural network and weighted (DCW) transformer model is proposed to address the problem of insufficient extraction of local information and no attention to channel‐step information in the traditional transformer model. First, a double‐channel information extraction method is proposed, so both the channel‐step and time‐step information achieve attention; second, the local information in the time and channel dimension of the data is extracted from deep and multiple scales, improving the feature extraction capability for local information; third, the long distance dependency relationship of the data is preserved by the attention mechanism, hence, the global correlation of the data is extracted effectively; finally, using the Gumbel‐SoftMax function, the weights of the time‐step and channel‐step feature information are assigned, so the extracted feature information has been optimized. The proposed method was applied for the penicillin fermentation process to verify its efficacy. Experimental results show that the proposed method achieved a better fault detection accuracy, outperforming the existing models. Further ablation experiments were conducted to demonstrate the effectiveness of each component of the proposed model.

TUMbRAIN: A transformer with a unified mobile residual attention inverted network for diagnosing brain tumors from magnetic resonance scans

Diagnosing tumors in Magnetic Resonance Imaging (MRI) brain scans is challenging and can lead to errors, even for radiologists. Deep learning, mainly through deep convolutional neural networks, has assisted in automating the diagnosis of these scans. However, there is still room for improvement. Researchers have shown that transformer models hold promise but often remain underutilized due to their need for large amounts of data and complexity compared to traditional neural networks. This paper introduces a new hybrid model called TUMbRAIN (Transformer with a Unified Mobile Residual Attention Inverted Network), which combines a lightweight transformer with a deep convolutional neural network to address these issues. TUMbRAIN incorporates innovative components designed for this purpose, such as the expanded inverted residual block and the unified self-attention mechanism. The results demonstrate that TUMbRAIN outperforms many existing state-of-the-art neural network models, achieving an impressive overall accuracy of 97.94 % with only 1.04 million parameters. These results suggest that hybrid transformer models like TUMbRAIN could significantly advance the automated diagnosis of brain tumors from MRI scans. The study also offers new insights into effectively integrating transformers into traditional neural network architectures, resulting in a cost-effective and accurate deep learning solution for medical imaging. By incorporating these advanced components, TUMbRAIN enhances support for radiological practice through improved diagnostic accuracy and efficiency.

A novel multi-label classification deep learning method for hybrid fault diagnosis in complex industrial processes

Hybrid Fault Detection and Diagnosis, encompassing both individual and simultaneous faults, is an important solution needed in chemical process safety practice. We systematically examine the two perspectives of simultaneous faults in previous studies: multi-class and multi-label classification, and highlight the limitations of the former while demonstrating the efficacy of the latter. Then, a novel multi-label classification Hybrid Fault Transformer (mcHFT) model was put forward to address hybrid faults. Our model is capable of learning not only intrinsic features of individual faults but also their coupled relationships. Importantly, this work constitutes the first comprehensive evaluation of Hybrid FDD on the Tennessee Eastman (TE) process to our knowledge. The mcHFT model significantly enhances key performance indicators over existing models and introduces an adaptive strategy to reduce false positives. The dataset developed for this research is made available under an MIT license, contributing a valuable resource for future exploration in this field.