Transformer Model Research Articles

Automatic delineation and prognostic assessment of head and neck tumor lesion in multi-modality positron emission tomography / computed tomography images based on deep learning: A survey

Accurately segmenting and staging tumor lesions in cancer patients presents a significant challenge for radiologists, but it is essential for devising effective treatment plans including radiation therapy, personalized medicine, and surgical options. The integration of artificial intelligence (AI), particularly deep learning (DL), has become a useful tool for radiologists, enhancing their ability to understand tumor biology and deliver personalized care to patients with H&N tumors. Segmenting H&N tumor lesions using Positron Emission Tomography/Computed Tomography (PET/CT) images has gained significant attention. However, the diverse shapes and sizes of tumors, along with indistinct boundaries between malignant and normal tissues, present significant challenges in effectively fusing PET and CT images. To overcome these challenges, various DL-based models have been developed for segmenting tumor lesions in PET/CT images. This article reviews multimodality (PET/CT) based H&N tumor lesions segmentation methods. We firstly discuss the strengths and limitations of PET/CT imaging and the importance of DL-based models in H&N tumor lesion segmentation. Second, we examine the current state-of-the-art DL models for H&N tumor segmentation, categorizing them into UNet, VNet, Vision Transformer, and miscellaneous models based on their architectures. Third, we explore the annotation and evaluation processes, addressing challenges in segmentation annotation and discussing the metrics used to assess model performance. Finally, we discuss several open challenges and provide some avenues for future research in H&N tumor lesion segmentation.

Topic modeling of neuropsychiatric diseases related to gut microbiota and gut brain axis using artificial intelligence based BERTopic model on PubMed abstracts

Gut microbiota play a crucial role in complex interactions of the gut brain axis between the gastrointestinal system and the central nervous system. The intricate network of bidirectional communication between the gut and brain, mediated through neural, hormonal, and immunological pathways, known as the gut-brain axis, has been implicated in the pathophysiology of several mental, neurological and behavioral disorders. Alterations in the gut microbiota composition, or dysbiosis, have been associated with disorders like Alzheimer's disease, Parkinson's disease, Multiple Sclerosis, Autism Spectrum Disorder, Ischemic Stroke, Eating Disorders, depression, anxiety, stress and addiction. In this study, a Python package BERTopic, based on Artificial Intelligence based Natural Language Processing using Transformer model BERT, specializing in topic modeling, was applied to abstracts of 3,482 PubMed articles published from year 2014 until May 2024, to explore the mental, neurological, and behavioral diseases influenced by the gut microbiota. There were some variations in individual runs of BERTopic due to stochastic nature of one of its components, but overall the discovered topics corresponded to major neuropsychiatric diseases. To understand the impact of the variability in outcomes ten repeated runs of BERTopic were performed with keeping identical parameters. The major topics that were found consistently in all the ten repeated runs of BERTopic were Depression, Alzheimer Disease, Autism Spectrum Disorder, Parkinson's Disease, Multiple Sclerosis, Ischemic Stroke, Anorexia Nervosa and Schizophrenia.

Data Augmentation in Histopathological Classification: An Analysis Exploring GANs with XAI and Vision Transformers

Generative adversarial networks (GANs) create images by pitting a generator (G) against a discriminator (D) network, aiming to find a balance between the networks. However, achieving this balance is difficult because G is trained based on just one value representing D’s prediction, and only D can access image features. We introduce a novel approach for training GANs using explainable artificial intelligence (XAI) to enhance the quality and diversity of generated images in histopathological datasets. We leverage XAI to extract feature information from D and incorporate it into G via the loss function, a unique strategy not previously explored in this context. We demonstrate that this approach enriches the training with relevant information and promotes improved quality and more variability in the artificial images, decreasing the FID by up to 32.7% compared to traditional methods. In the data augmentation task, these images improve the classification accuracy of Transformer models by up to 3.81% compared to models without data augmentation and up to 3.01% compared to traditional GAN data augmentation. The Saliency method provides G with the most informative feature information. Overall, our work highlights the potential of XAI for enhancing GAN training and suggests avenues for further exploration in this field.

Multi-modal transformer with language modality distillation for early pedestrian action anticipation

Language-vision integration has become an increasingly popular research direction within the computer vision field. In recent years, there has been a growing recognition of the importance of incorporating linguistic information into visual tasks, particularly in domains such as action anticipation. This integration allows anticipation models to leverage textual descriptions to gain deeper contextual understanding, leading to more accurate predictions. In this work, we focus on pedestrian action anticipation, where the objective is the early prediction of pedestrians’ future actions in urban environments. Our method relies on a multi-modal transformer model that encodes past observations and produces predictions at different anticipation times, employing a learned mask technique to filter out redundancy in the observed frames. Instead of relying solely on visual cues extracted from images or videos, we explore the impact of integrating textual information in enriching the input modalities of our pedestrian action anticipation model. We investigate various techniques for generating descriptive captions corresponding to input images, aiming to enhance the anticipation performance. Evaluation results on available public benchmarks demonstrate the effectiveness of our method in improving the prediction performance at different anticipation times compared to previous works. Additionally, incorporating the language modality in our anticipation model proved significant improvement, reaching a 29.5% increase in the F1 score at 1-second anticipation and a 16.66% increase at 4-second anticipation. These results underscore the potential of language-vision integration in advancing pedestrian action anticipation in complex urban environments.

Optimized array for powering sensors: Thermal and electromagnetic energy harvesting and fusion

Environmental energy harvesting technology has achieved great success in remote sensor power supply applications. However, existing technologies in the power grid neglect the potential of weak energy sources. This study aims to optimize the most reasonable array configurations and create hybrid energy fusion schemes for different weak energy environments. Taking thermal and electromagnetic energy harvesting from cable surfaces and currents in coaxial cables as an example, this paper introduces a genetic algorithm-based optimization method of harvester array configuration under multiple constraints and a dual-source energy fusion system. First, a vector impedance parameter identification method based on the equivalent circuit model of current transformer (CT) is proposed. Later, an optimization model for harvester array is established. Then, a genetic algorithm is used to optimize array combinations. And, a dual-source energy fusion system is proposed. Finally, thermal and electromagnetic energy harvesting and fusion are performed using 3 × 1 TEGs array and 6 × 1CT array. Under the specified conditions of a hot end temperature of 50 ℃ and an effective current value of 2 A, the power supply for the Zigbee wireless sensing system is realized to monitor the temperature of the cable core.

A coordinated adaptive multiscale enhanced spatio-temporal fusion network for multi-lead electrocardiogram arrhythmia detection

The multi-lead electrocardiogram (ECG) is widely utilized in clinical diagnosis and monitoring of cardiac conditions. The advancement of deep learning has led to the emergence of automated multi-lead ECG diagnostic networks, which have become essential in the fields of biomedical engineering and clinical cardiac disease diagnosis. Intelligent ECG diagnosis techniques encompass Recurrent Neural Networks (RNN), Transformers, and Convolutional Neural Networks (CNN). While CNN is capable of extracting local spatial information from images, it lacks the ability to learn global spatial features and temporal memory features. Conversely, RNN relies on time and can retain significant sequential features. However, they are not proficient in extracting lengthy dependencies of sequence data in practical scenarios. The self-attention mechanism in the Transformer model has the capability of global feature extraction, but it does not adequately prioritize local features and cannot extract spatial and channel features. This paper proposes STFAC-ECGNet, a model that incorporates CAMV-RNN block, CBMV-CNN block, and TSEF block to enhance the performance of the model by integrating the strengths of CNN, RNN, and Transformer. The CAMV-RNN block incorporates a coordinated adaptive simplified self-attention module that adaptively carries out global sequence feature retention and enhances spatial–temporal information. The CBMV-CNN block integrates spatial and channel attentional mechanism modules in a skip connection, enabling the fusion of spatial and channel information. The TSEF block implements enhanced multi-scale fusion of image spatial and sequence temporal features. In this study, comprehensive experiments were conducted using the PTB-XL large publicly available ECG dataset and the China Physiological Signal Challenge 2018 (CPSC2018) database. The results indicate that STFAC-ECGNet surpasses other cutting-edge techniques in multiple tasks, showcasing robustness and generalization.

Behavior of LSTM and Transformer deep learning models in flood simulation considering South Asian tropical climate

ABSTRACT The imperative for a reliable and accurate flood forecasting procedure stems from the hazardous nature of the disaster. In response, researchers are increasingly turning to innovative approaches, particularly machine learning models, which offer enhanced accuracy compared to traditional methods. However, a notable gap exists in the literature concerning studies focused on the South Asian tropical region, which possesses distinct climate characteristics. This study investigates the applicability and behavior of long short-term memory (LSTM) and transformer models in flood simulation considering the Mahaweli catchment in Sri Lanka, which is mostly affected by the Northeast Monsoon. The importance of different input variables in the prediction was also a key focus of this study. Input features for the models included observed rainfall data collected from three nearby rain gauges, as well as historical discharge data from the target river gauge. Results showed that the use of past water level data denotes a higher impact on the output compared to the other input features such as rainfall, for both architectures. All models denoted satisfactory performances in simulating daily water levels, with Nash–Sutcliffe Efficiency (NSE) values greater than 0.77 while the transformer encoder model showed a superior performance compared to encoder–decoder models.

Unraveling the key role of chromatin structure in cancer development through epigenetic landscape characterization of oral cancer

Epigenetic alterations, such as those in chromatin structure and DNA methylation, have been extensively studied in a number of tumor types. But oral cancer, particularly oral adenocarcinoma, has received far less attention. Here, we combined laser-capture microdissection and muti-omics mini-bulk sequencing to systematically characterize the epigenetic landscape of oral cancer, including chromatin architecture, DNA methylation, H3K27me3 modification, and gene expression. In carcinogenesis, tumor cells exhibit reorganized chromatin spatial structures, including compromised compartment structures and altered gene-gene interaction networks. Notably, some structural alterations are observed in phenotypically non-malignant paracancerous but not in normal cells. We developed transformer models to identify the cancer propensity of individual genome loci, thereby determining the carcinogenic status of each sample. Insights into cancer epigenetic landscapes provide evidence that chromatin reorganization is an important hallmark of oral cancer progression, which is also linked with genomic alterations and DNA methylation reprogramming. In particular, regions of frequent copy number alternations in cancer cells are associated with strong spatial insulation in both cancer and normal samples. Aberrant methylation reprogramming in oral squamous cell carcinomas is closely related to chromatin structure and H3K27me3 signals, which are further influenced by intrinsic sequence properties. Our findings indicate that structural changes are both significant and conserved in two distinct types of oral cancer, closely linked to transcriptomic alterations and cancer development. Notably, the structural changes remain markedly evident in oral adenocarcinoma despite the considerably lower incidence of genomic copy number alterations and lesser extent of methylation alterations compared to squamous cell carcinoma. We expect that the comprehensive analysis of epigenetic reprogramming of different types and subtypes of primary oral tumors can provide additional guidance to the design of novel detection and therapy for oral cancer.

Intelligent tool wear prediction based on deep learning PSD-CVT model

To ensure the reliability of machining quality, it is crucial to predict tool wear accurately. In this paper, a novel deep learning-based model is proposed, which synthesizes the advantages of power spectral density (PSD), convolutional neural networks (CNN), and vision transformer model (ViT), namely PSD-CVT. PSD maps can provide a comprehensive understanding of the spectral characteristics of the signals. It makes the spectral characteristics more obvious and makes it easy to analyze and compare different signals. CNN focuses on local feature extraction, which can capture local information such as the texture, edge, and shape of the image, while the attention mechanism in ViT can effectively capture the global structure and long-range dependencies present in the image. Two fully connected layers with a ReLU function are used to obtain the predicted tool wear values. The experimental results on the PHM 2010 dataset demonstrate that the proposed model has higher prediction accuracy than the CNN model or ViT model alone, as well as outperforms several existing methods in accurately predicting tool wear. The proposed prediction method can also be applied to predict tool wear in other machining fields.

Vision transformers inference acceleration based on adaptive layer normalization

Over the past decade, deep neural networks (DNNs) have been widely used due to their remarkable accuracy in real-world applications. However, this increase in accuracy often results in computationally expensive models and high memory usage, leading to longer prediction latencies and exorbitant release costs. In this paper, we propose a new adaptive layer normalization (ALN) algorithm for transformer models, which tackles the computational, and memory problems encountered by traditional layer normalization (LN) techniques. The proposed method computes and stores statistical moments during the training and uses them directly during the inference phase, allowing the normalization layer to be merged with the nearest linear layer. The result is a significant acceleration in inference time, by up to 29%. In classification tasks, our evaluations on the ImageNet dataset show an improvement in accuracy of 0.1%, while maintaining comparable accuracy in object detection tasks on the COCO reference dataset. The proposed ALN algorithm is a simple and effective solution for improving the inference time of pre-trained transformer models, making it a valuable tool for natural language processing and computer vision tasks.

Residual Dense Optimization-Based Multi-Attention Transformer to Detect Network Intrusion against Cyber Attacks

Achieving cyber-security has grown increasingly tricky because of the rising concern for internet connectivity and the significant growth in software-related applications. It also needs a robust defense system to defend itself from multiple cyberattacks. Therefore, there is a need to generate a method for detecting and classifying cyber-attacks. The developed model can be integrated into three phases: pre-processing, feature selection, and classification. Initially, the min-max normalization of original data was performed to eliminate the impact of maximum or minimum values on the overall characteristics. After that, synthetic minority oversampling techniques (SMOTEs) were developed to reduce the number of minority attacks. The significant features were selected using a Hybrid Genetic Fire Hawk Optimizer (HGFHO). An optimized residual dense-assisted multi-attention transformer (Op-ReDMAT) model was introduced to classify selected features accurately. The proposed model’s performance was evaluated using the UNSW-NB15 and CICIDS2017 datasets. A performance analysis was carried out to demonstrate the effectiveness of the proposed model. The experimental results showed that the UNSW-NB15 dataset attained a higher precision, accuracy, F1-score, error rate, and recall of 97.2%, 98.82%, 97.8%, 2.58, and 98.5%, respectively. On the other hand, the CICIDS 2017 achieved a higher precision, accuracy, F1-score, and recall of 98.6%, 99.12%, 98.8%, and 98.2%, respectively.

CACER: Clinical concept Annotations for Cancer Events and Relations.

Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. We present Clinical concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48000 medical problems and drug events and 10000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction models such as Bidirectional Encoder Representations from Transformers (BERT), Fine-tuned Language Net Text-To-Text Transfer Transformer (Flan-T5), Large Language Model Meta AI (Llama3), and Generative Pre-trained Transformers-4 (GPT-4) using fine-tuning and in-context learning (ICL). In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks. The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, large language models offer no performance advantage over the smaller BERT models. We introduce CACER, a novel corpus with fine-grained annotations for medical problems, drugs, and their relationships in clinical narratives of oncology notes. State-of-the-art transformer models achieved performance comparable to IAA for several extraction tasks.

EC-Conf: A ultra-fast diffusion model for molecular conformation generation with equivariant consistency

Despite recent advancement in 3D molecule conformation generation driven by diffusion models, its high computational cost in iterative diffusion/denoising process limits its application. Here, an equivariant consistency model (EC-Conf) was proposed as a fast diffusion method for low-energy conformation generation. In EC-Conf, a modified SE (3)-equivariant transformer model was directly used to encode the Cartesian molecular conformations and a highly efficient consistency diffusion process was carried out to generate molecular conformations. It was demonstrated that, with only one sampling step, it can already achieve comparable quality to other diffusion-based models running with thousands denoising steps. Its performance can be further improved with a few more sampling iterations. The performance of EC-Conf is evaluated on both GEOM-QM9 and GEOM-Drugs sets. Our results demonstrate that the efficiency of EC-Conf for learning the distribution of low energy molecular conformation is at least two magnitudes higher than current SOTA diffusion models and could potentially become a useful tool for conformation generation and sampling.Scientific ContributionsIn this work, we proposed an equivariant consistency model that significantly improves the efficiency of conformation generation in diffusion-based models while maintaining high structural quality. This method serves as a general framework and can be further extended to more complex structure generation and prediction tasks, including those involving proteins, in future steps.

A new approach towards more accurate modeling of mechanical defects in power transformer windings

In many cases, power transformers can continue their normal operation despite mechanical defects in their windings. However, estimating the lifespan and assessing the risk of using a transformer with mechanical defects depend on detailed studies of insulation aging and the distribution of transient voltages along the winding, and consequently, the likelihood of partial discharge. Risk assessment studies can only be effectively conducted through accurate modeling of transformers across a wide frequency range. In this study, an attempt has been made to investigate the impact of winding mechanical defects on the transformer equivalent circuits parameters using finite element method precisely modeling of transformer windings, without applying approximation or common simplifications. To compare the effect of each mechanical defect on the resistance, inductance, and capacitance matrices arrays, an index called fault classification accuracy has been used. The investigations have shown that the capacitance matrix arrays undergo the most significant changes for all types of modeled defects in this research. To examine the impact of defects on the transformer frequency response, a π coupled model that directly uses output matrices of finite element modeling has been employed. The sensitivity of various components of frequency response has also been investigated for the used model.

AMTransformer: A Koopman theory-based transformer for learning additive manufacturing dynamics in laser processes

Recent advancements in machine learning (ML) have shown unprecedented promise in understanding and predicting additive manufacturing (AM) dynamics. However, existing ML studies on AM often lack a comprehensive approach to address the multi-scale complexities inherent in AM processes and tend to employ context-specific methods. To address these limitations, we present a foundational method for formulating AM dynamics suitable for ML modeling. We then introduce a novel approach, the AMTransformer, designed to comprehend complex spatiotemporal dynamical dependencies among physical entities and their properties within the AM process. To enhance the understanding of AM dynamics, our method adapts Koopman’s theory to generate latent embeddings of AM states and their transitions, effectively extracting hidden features related to physical properties and dynamical dependencies. In addition, by utilizing the transformer’s attention mechanism, the proposed approach enhances the learning of non-local, non-linear dynamical dependencies across multiple scales. Our experiments, conducted using melt pool data from a laser powder bed fusion process, demonstrate that the AMTransformer outperforms traditional transformer and convolutional long short-term memory models. Specifically, the AMTransformer achieved structural similarity, mean absolute error, and accuracy metric values of 0.9206, 0.0009 mm2, and 92.73%, respectively. These results indicate the AMTransformer’s superior ability to predict future AM states, attributed to its improved learning of complex AM dynamics. By combining linear Koopman-based methods with non-linear transformer-based approaches, the AMTransformer significantly improves data-driven modeling for AM, providing a more comprehensive understanding of AM dynamics. Furthermore, the generalizability of the proposed method facilitates the expansion of the model’s scope and enhances its applicability across various fields.

A 3D-stack DRAM-based PNM architecture design

The article examines methods for integrating 3D-stacked DRAM with AI logic chips, in order to overcome the memory bandwidth challenges faced in the AI inference of transformer models. The findings indicate that this approach can yield a 9x to 3x reduction in power consumption while maintaining similar performance levels, or alternatively, an 8x improvement in performance with comparable power consumption.

Short-Term Load Forecasting for Regional Smart Energy Systems Based on Two-Stage Feature Extraction and Hybrid Inverted Transformer

Accurate short-term load forecasting is critical for enhancing the reliability and stability of regional smart energy systems. However, the inherent challenges posed by the substantial fluctuations and volatility in electricity load patterns necessitate the development of advanced forecasting techniques. In this study, a novel short-term load forecasting approach based on a two-stage feature extraction process and a hybrid inverted Transformer model is proposed. Initially, the Prophet method is employed to extract essential features such as trends, seasonality and holiday patterns from the original load dataset. Subsequently, variational mode decomposition (VMD) optimized by the IVY algorithm is utilized to extract significant periodic features from the residual component obtained by Prophet. The extracted features from both stages are then integrated to construct a comprehensive data matrix. This matrix is then inputted into a hybrid deep learning model that combines an inverted Transformer (iTransformer), temporal convolutional networks (TCNs) and a multilayer perceptron (MLP) for accurate short-term load forecasting. A thorough evaluation of the proposed method is conducted through four sets of comparative experiments using data collected from the Elia grid in Belgium. Experimental results illustrate the superior performance of the proposed approach, demonstrating high forecasting accuracy and robustness, highlighting its potential in ensuring the stable operation of regional smart energy systems.

SpemNet: A Cotton Disease and Pest Identification Method Based on Efficient Multi-Scale Attention and Stacking Patch Embedding.

We propose a cotton pest and disease recognition method, SpemNet, based on efficient multi-scale attention and stacking patch embedding. By introducing the SPE module and the EMA module, we successfully solve the problems of local feature learning difficulty and insufficient multi-scale feature integration in the traditional Vision Transformer model, which significantly improve the performance and efficiency of the model. In our experiments, we comprehensively validate the SpemNet model on the CottonInsect dataset, and the results show that SpemNet performs well in the cotton pest recognition task, with significant effectiveness and superiority. The SpemNet model excels in key metrics such as precision and F1 score, demonstrating significant potential and superiority in the cotton pest and disease recognition task. This study provides an efficient and reliable solution in the field of cotton pest and disease identification, which is of great theoretical and applied significance.

Multi-scale convolution enhanced transformer for multivariate long-term time series forecasting

In data analysis and forecasting, particularly for multivariate long-term time series, challenges persist. The Transformer model in deep learning methods has shown significant potential in time series forecasting. The Transformer model’s dot-product attention mechanism, however, due to its quadratic computational complexity, impairs training and forecasting efficiency. In addition, the Transformer architecture has limitations in modeling local features and dealing with multivariate cross-dimensional dependency relationship. In this article, a Multi-Scale Convolution Enhanced Transformer model (MSCformer) is proposed for multivariate long-term time series forecasting. As an alternative to modeling the time series in its entirety, a segmentation strategy is designed to convert the input original series into segmented forms with different lengths, then process time series segments using a new constructed multi-Dependency Aggregation module. This multi-Scale segmentation approach reduces the computational complexity of the attention mechanism part in subsequent models, and for each segment of length corresponds to a specific time scale, it also ensures that each segment retains the semantic information of the data sequence level, thereby comprehensively utilizing the multi-scale information of the data while more accurately capturing the real dependency of the time series. The Multi-Dependence Aggregate module captures both cross-temporal and cross-dimensional dependencies of multivariate long-term time series and compensates for local dependencies within the segments thereby captures local series features comprehensively and addressing the issue of insufficient information utilization. MSCformer synthesizes dependency information extracted from various temporal segments at different scales and reconstructs future series using linear layers. MSCformer exhibits higher forecasting accuracy, outperforming existing methods in multiple domains including energy, transportation, weather, electricity, disease and finance.

TDFFM: Transformer and Deep Forest Fusion Model for Predicting Coronavirus 3C-Like Protease Cleavage Sites.

COVID-19, caused by the highly contagious SARS-CoV-2 virus, is distinguished by its positive-sense, single-stranded RNA genome. A thorough understanding of SARS-CoV-2 pathogenesis is crucial for halting its proliferation. Notably, the 3C-like protease of the coronavirus (denoted as 3CLpro) is instrumental in the viral replication process. Precise delineation of 3CLpro cleavage sites is imperative for elucidating the transmission dynamics of SARS-CoV-2. While machine learning tools have been deployed to identify potential 3CLpro cleavage sites, these existing methods often fall short in terms of accuracy. To improve the performances of these predictions, we propose a novel analytical framework, the Transformer and Deep Forest Fusion Model (TDFFM). Within TDFFM, we utilize the AAindex and the BLOSUM62 matrix to encode protein sequences. These encoded features are subsequently input into two distinct components: a Deep Forest, which is an effective decision tree ensemble methodology, and a Transformer equipped with a Multi-Level Attention Model (TMLAM). The integration of the attention mechanism allows our model to more accurately identify positive samples, thus enhancing the overall predictive performance. Evaluation on a test set demonstrates that our TDFFM achieves an accuracy of 0.955, an AUC of 0.980, and an F1-score of 0.367, substantiating the model's superior prediction capabilities.