Transformer-based Approach Research Articles

LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension

Incorporating prior knowledge has been identified as a promising approach to enhance existing pre-training models in cloze-style machine reading, as observed in recent studies. Despite the use of external knowledge graphs (KG) and transformer-based models like BERT in most existing models, the identification of the most pertinent ambiguous entities in KG, as well as the extraction of the optimal subgraphs, remains problematic. To address these challenges, we introduce the LUKE-Graph model, which constructs a heterogeneous graph based on the intuitive relationships between entities in the documents without relying on external KGs. We then employ a Relational Graph Attention (RGAT) network to combine the reasoning information of the graph with the contextual representation generated by the pre-trained LUKE model. In this way, we can take advantage of LUKE, to derive an entity-aware representation; and a graph model - to exploit relation-aware representation. Furthermore, we present Gated-RGAT, an enhancement to RGAT that incorporates a gating mechanism to control the question information during the graph convolution operation. This mechanism emulates the human reasoning process in selecting the most suitable entity candidate based on question information. Our experimental results demonstrate that the proposed LUKE-Graph model surpasses the LUKE state-of-the-art model on the ReCoRD dataset, which focuses on commonsense reasoning, and the WikiHop dataset, which centers on multi-hop reasoning problems.

Assessing the Performance of Transformer for Time Series Anomaly Detection

This study aims to assess the effectiveness of the Transformer-based reconstruction approach for detecting anomalies in time series data. The reconstruction error-based anomaly detection method was applied to both multivariate time series from NASA SMAP/MSL and univariate time series from UCR. Four deep learning models, including Transformer, Dilated CNN, LSTM, and MLP, were compared in terms of their ability to reconstruct input data. Dilated CNN outperformed the other models in almost all experimental results, achieving a 25% higher score than Transformer on the UCR dataset when trained with random masking, and a 60% higher score when trained with middle masking. These results suggest that the Transformer did not perform as well as expected for anomaly detection based on time series reconstruction errors, and its inferiority to Dilated CNN may be attributed to the characteristics of the time series and the limited training data. Future research should focus on developing Transformer models that can better capture the properties of time series data and investigate the relationship between the model’s performance, data volume, and model complexity.

Dynamically Relative Position Encoding-Based Transformer for Automatic Code Edit

Adapting deep learning (DL) techniques to automate nontrivial coding activities, such as code documentation and defect detection, has been intensively studied recently. Learning to predict code changes is one of the popular and essential investigations. Prior studies have shown that DL techniques, such as neural machine translation (NMT), can benefit meaningful code changes, including bug fixing and code refactoring. However, NMT models may encounter bottleneck when modeling long sequences; thus, they are limited in accurately predicting code changes. In this article, we design a Transformer-based approach, considering that the Transformer has proven effective in capturing long-term dependencies. Specifically, we propose a novel model named DTrans. For better incorporating the local structure of code, i.e., statement-level information in this article, DTrans is designed with dynamically relative position encoding in the multihead attention of the Transformer. Experiments on benchmark datasets demonstrate that DTrans can more accurately generate patches than the state-of-the-art methods, increasing the performance by at least 5.45–46.57% in terms of the exact match metric on different datasets. Moreover, DTrans can locate the lines to change with 1.75–24.21% higher accuracy than the existing methods.

Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study

Deep learning (DL) can accelerate the prediction of prognostic biomarkers from routine pathology slides in colorectal cancer (CRC). However, current approaches rely on convolutional neural networks (CNNs) and have mostly been validated on small patient cohorts. Here, we develop a new transformer-based pipeline for end-to-end biomarker prediction from pathology slides by combining a pre-trained transformer encoder with a transformer network for patch aggregation. Our transformer-based approach substantially improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training and evaluating on a large multicenter cohort of over 13,000 patients from 16 colorectal cancer cohorts, we achieve a sensitivity of 0.99 with a negative predictive value of over 0.99 for prediction of microsatellite instability (MSI) on surgical resection specimens. We demonstrate that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem.

Hybrid feature integration model and adaptive transformer approach for emotion recognition with EEG signals

Nowadays, electroencephalogram (EEG) signals are the major part of recognizing emotions as the signals contain more brain-related information. Several traditional recognition methodologies have been developed, but still, exist with certain challenges that degrade and mislead the recognition process. Some models are facing issues based on insignificant features, poor generalization capability, and computational burden. To alleviate such issues, an effective emotion recognition model is proposed using a heuristic-aided transformer-based approach. Initially, the input EEG signals are gathered and decomposed via the 5-level Discrete Wavelet Transform (DWT). Then, the decomposed signals are fed to the feature extraction process, where the feature extraction is carried out by using Logistic Regression (LR) approach. From the decomposed signals, the Logistic Regression (LR) is used for getting the noteworthy features and then, the same decomposed signals are fed to the optimal weighted feature selection, which is carried out using Adaptive Bald Eagle Search Optimization (ABESO) algorithm. Subsequently, the hybrid weighted feature selection approach is introduced for getting the weighted features, where the weight is optimized with the adoption of the ABESO algorithm. At last, the selected hybrid weighted features are sent to the recognition phase, where the Optimized Block Recurrent Transformer (OBRT), where performance gets enhanced by tuning the parameter with the developed ABESO algorithm. The accuracy and precision rate of the designed approach are attained as 96% and 94%. The performance of the model is assessed and the findings have shown that better performance that makes the system more feasible to recognize human emotions.

Temporal–spatial transformer based motor imagery classification for BCI using independent component analysis

Motor Imagery (MI) classification with electroencephalography (EEG) is a critical aspect of Brain–Computer Interface (BCI) systems, enabling individuals with mobility limitations to communicate with the outside world. However, the complexity, variability, and low signal-to-noise ratio of EEG data present significant challenges in decoding these signals, particularly in a subject-independent manner. To overcome these challenges, we propose a transformer-based approach that employs a self-attention process to extract features in the temporal and spatial domains. To establish spatial correlations across MI EEG channels, the self-attention module periodically updates each channel by averaging its features across all channels. This weighted averaging improves classification accuracy and removes artifacts generated by manually selecting channels. Furthermore, the temporal self-attention mechanism encodes global sequential information into the features for each sample time step, allowing for the extraction of superior temporal properties in the time domain from MI EEG data. The effectiveness of the proposed strategy has been confirmed through testing against the BCI Competition IV 2a and 2b benchmarks. Overall, our proposed model outperforms state-of-the-art methods and demonstrates greater stability in both subject-dependent and subject-independent strategies.

Local and global convolutional transformer-based motor imagery EEG classification.

Transformer, a deep learning model with the self-attention mechanism, combined with the convolution neural network (CNN) has been successfully applied for decoding electroencephalogram (EEG) signals in Motor Imagery (MI) Brain-Computer Interface (BCI). However, the extremely non-linear, nonstationary characteristics of the EEG signals limits the effectiveness and efficiency of the deep learning methods. In addition, the variety of subjects and the experimental sessions impact the model adaptability. In this study, we propose a local and global convolutional transformer-based approach for MI-EEG classification. The local transformer encoder is combined to dynamically extract temporal features and make up for the shortcomings of the CNN model. The spatial features from all channels and the difference in hemispheres are obtained to improve the robustness of the model. To acquire adequate temporal-spatial feature representations, we combine the global transformer encoder and Densely Connected Network to improve the information flow and reuse. To validate the performance of the proposed model, three scenarios including within-session, cross-session and two-session are designed. In the experiments, the proposed method achieves up to 1.46%, 7.49% and 7.46% accuracy improvement respectively in the three scenarios for the public Korean dataset compared with current state-of-the-art models. For the BCI competition IV 2a dataset, the proposed model also achieves a 2.12% and 2.21% improvement for the cross-session and two-session scenarios respectively. The results confirm that the proposed approach can effectively extract much richer set of MI features from the EEG signals and improve the performance in the BCI applications.

DBAFormer: A Double-Branch Attention Transformer for Long-Term Time Series Forecasting

The transformer-based approach excels in long-term series forecasting. These models leverage stacking structures and self-attention mechanisms, enabling them to effectively model dependencies in series data. While some approaches prioritize sparse attention to tackle the quadratic time complexity of self-attention, it can limit information utilization. We introduce a creative double-branch attention mechanism that simultaneously captures intricate dependencies in both temporal and variable perspectives. Moreover, we propose query-independent attention, taking into account the near-identical attention allocated by self-attention to different query positions. This enhances efficiency and reduces the impact of redundant information. We integrate the double-branch query-independent attention into popular transformer-based methods like Informer, Autoformer, and Non-stationary transformer. The results obtained from conducting experiments on six practical benchmarks consistently validate that our novel attention mechanism substantially improves the long-term series forecasting performance in contrast to the baseline approach.

SMBCNet: A Transformer-Based Approach for Change Detection in Remote Sensing Images through Semantic Segmentation

Remote sensing change detection (RSCD) is crucial for our understanding of the dynamic pattern of the Earth’s surface and human influence. Recently, transformer-based methodologies have advanced from their powerful global modeling capabilities in RSCD tasks. Nevertheless, they remain under excessive parameterization, which continues to be severely constrained by time and computation resources. Here, we present a transformer-based RSCD model called the Segmentation Multi-Branch Change Detection Network (SMBCNet). Our proposed approach combines a hierarchically structured transformer encoder with a cross-scale enhancement module (CEM) to extract global information with lower complexity. To account for the diverse nature of changes, we introduce a plug-and-play multi-branch change fusion module (MCFM) that integrates temporal features. Within this module, we transform the change detection task into a semantic segmentation problem. Moreover, we identify the Temporal Feature Aggregation Module (TFAM) to facilitate integrating features from diverse spatial scales. These results demonstrate that semantic segmentation is an effective solution to change detection (CD) problems in remote sensing images.

Transformers for extracting breast cancer information from Spanish clinical narratives

The wide adoption of electronic health records (EHRs) offers immense potential as a source of support for clinical research. However, previous studies focused on extracting only a limited set of medical concepts to support information extraction in the cancer domain for the Spanish language. Building on the success of deep learning for processing natural language texts, this paper proposes a transformer-based approach to extract named entities from breast cancer clinical notes written in Spanish and compares several language models. To facilitate this approach, a schema for annotating clinical notes with breast cancer concepts is presented, and a corpus for breast cancer is developed. Results indicate that both BERT-based and RoBERTa-based language models demonstrate competitive performance in clinical Named Entity Recognition (NER). Specifically, BETO and multilingual BERT achieve F-scores of 93.71% and 94.63%, respectively. Additionally, RoBERTa Biomedical attains an F-score of 95.01%, while RoBERTa BNE achieves an F-score of 94.54%. The findings suggest that transformers can feasibly extract information in the clinical domain in the Spanish language, with the use of models trained on biomedical texts contributing to enhanced results. The proposed approach takes advantage of transfer learning techniques by fine-tuning language models to automatically represent text features and avoiding the time-consuming feature engineering process.

HyperSFormer: A Transformer-Based End-to-End Hyperspectral Image Classification Method for Crop Classification

Crop classification of large-scale agricultural land is crucial for crop monitoring and yield estimation. Hyperspectral image classification has proven to be an effective method for this task. Most current popular hyperspectral image classification methods are based on image classification, specifically on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In contrast, this paper focuses on methods based on semantic segmentation and proposes a new transformer-based approach called HyperSFormer for crop hyperspectral image classification. The key enhancement of the proposed method is the replacement of the encoder in SegFormer with an improved Swin Transformer while keeping the SegFormer decoder. The entire model adopts a simple and uniform transformer architecture. Additionally, the paper introduces the hyper patch embedding (HPE) module to extract spectral and local spatial information from the hyperspectral images, which enhances the effectiveness of the features used as input for the model. To ensure detailed model processing and achieve end-to-end hyperspectral image classification, the transpose padding upsample (TPU) module is proposed for the model’s output. In order to address the problem of insufficient and imbalanced samples in hyperspectral image classification, the paper designs an adaptive min log sampling (AMLS) strategy and a loss function that incorporates dice loss and focal loss to assist model training. Experimental results using three public hyperspectral image datasets demonstrate the strong performance of HyperSFormer, particularly in the presence of imbalanced sample data, complex negative samples, and mixed sample classes. HyperSFormer outperforms state-of-the-art methods, including fast patch-free global learning (FPGA), a spectral–spatial-dependent global learning framework (SSDGL), and SegFormer, by at least 2.7% in the mean intersection over union (mIoU). It also improves the overall accuracy and average accuracy values by at least 0.9% and 0.3%, respectively, and the kappa coefficient by at least 0.011. Furthermore, ablation experiments were conducted to determine the optimal hyperparameter and loss function settings for the proposed method, validating the rationality of these settings and the fusion loss function.

ITran: A novel transformer-based approach for industrial anomaly detection and localization

Anomaly detection is currently an essential quality monitoring process in industrial production. It is often affected by factors such as under or over reconstruction of images and unclear criteria for feature distribution evaluation, thus making it challenging to improve detection performance. To solve the above problems, this paper proposes a novel transformer-based approach, Inductive Transformer (ITran) for industrial anomaly detection and localization, which utilizes a multi-layer pyramid structure and multi-level jump connections to extract multi-scale features of the data, putting the anomaly detection into the feature space and achieving more accurate industrial anomaly detection and localization results. It incorporates inductive bias and convolution operations into the Transformer which helps to break the myth of Transformer being “data hungry”. Compared with the common Transformers, ITran significantly reduces the computational cost and memory usage and makes it work well on small datasets. In addition, we basically eliminate the effect of positional embedding on the proposed Transformer model. Sufficient experiments have been conducted to validate global anomaly detection on three datasets MNIST, Fashion-MNST and Cifar-10, as well as local anomaly detection on the industrial datasets MVTec AD, Concrete Crack Image and BTAD. The proposed ITran achieves outstanding results on all the above datasets.

A transformer-based approach for early prediction of soybean yield using time-series images.

Crop yield prediction which provides critical information for management decision-making is of significant importance in precision agriculture. Traditional manual inspection and calculation are often laborious and time-consuming. For yield prediction using high-resolution images, existing methods, e.g., convolutional neural network, are challenging to model long range multi-level dependencies across image regions. This paper proposes a transformer-based approach for yield prediction using early-stage images and seed information. First, each original image is segmented into plant and soil categories. Two vision transformer (ViT) modules are designed to extract features from each category. Then a transformer module is established to deal with the time-series features. Finally, the image features and seed features are combined to estimate the yield. A case study has been conducted using a dataset that was collected during the 2020 soybean-growing seasons in Canadian fields. Compared with other baseline models, the proposed method can reduce the prediction error by more than 40%. The impact of seed information on predictions is studied both between models and within a single model. The results show that the influence of seed information varies among different plots but it is particularly important for the prediction of low yields.

A Transformer-Based Approach to Authorship Attribution in Classical Arabic Texts

Authorship attribution (AA) is a field of natural language processing that aims to attribute text to its author. Although the literature includes several studies on Arabic AA in general, applying AA to classical Arabic texts has not gained similar attention. This study focuses on investigating recent Arabic pretrained transformer-based models in a rarely studied domain with limited research contributions: the domain of Islamic law. We adopt an experimental approach to investigate AA. Because no dataset has been designed specifically for this task, we design and build our own dataset using Islamic law digital resources. We conduct several experiments on fine-tuning four Arabic pretrained transformer-based models: AraBERT, AraELECTRA, ARBERT, and MARBERT. Results of the experiments indicate that for the task of attributing a given text to its author, ARBERT and AraELECTRA outperform the other models with an accuracy of 96%. We conclude that pretrained transformer models, specifically ARBERT and AraELECTRA, fine-tuned using the Islamic legal dataset, show significant results in applying AA to Islamic legal texts.

Transformer-Based Approach to Melanoma Detection.

Melanoma is a malignant cancer type which develops when DNA damage occurs (mainly due to environmental factors such as ultraviolet rays). Often, melanoma results in intense and aggressive cell growth that, if not caught in time, can bring one toward death. Thus, early identification at the initial stage is fundamental to stopping the spread of cancer. In this paper, a ViT-based architecture able to classify melanoma versus non-cancerous lesions is presented. The proposed predictive model is trained and tested on public skin cancer data from the ISIC challenge, and the obtained results are highly promising. Different classifier configurations are considered and analyzed in order to find the most discriminating one. The best one reached an accuracy of 0.948, sensitivity of 0.928, specificity of 0.967, and AUROC of 0.948.

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

Zero-shot detection (ZSD) aims to locate and classify unseen objects in pictures or videos by semantic auxiliary information without additional training examples. Most of the existing ZSD methods are based on two-stage models, which achieve the detection of unseen classes by aligning object region proposals with semantic embeddings. However, these methods have several limitations, including poor region proposals for unseen classes, lack of consideration of semantic representations of unseen classes or their inter-class correlations, and domain bias towards seen classes, which can degrade overall performance. To address these issues, the Trans-ZSD framework is proposed, which is a transformer-based multi-scale contextual detection framework that explicitly exploits inter-class correlations between seen and unseen classes and optimizes feature distribution to learn discriminative features. Trans-ZSD is a single-stage approach that skips proposal generation and performs detection directly, allowing the encoding of long-term dependencies at multiple scales to learn contextual features while requiring fewer inductive biases. Trans-ZSD also introduces a foreground-background separation branch to alleviate the confusion of unseen classes and backgrounds, contrastive learning to learn inter-class uniqueness and reduce misclassification between similar classes, and explicit inter-class commonality learning to facilitate generalization between related classes. Trans-ZSD addresses the domain bias problem in end-to-end generalized zero-shot detection (GZSD) models by using balance loss to maximize response consistency between seen and unseen predictions, ensuring that the model does not bias towards seen classes. The Trans-ZSD framework is evaluated on the PASCAL VOC and MS COCO datasets, demonstrating significant improvements over existing ZSD models.

Local Attention Transformer-Based Full-View Finger-Vein Identification

Multi-view finger-vein recognition technology has attracted increasing attentions in recent years. Despite recent advances in the multi-view finger-vein identification, existing solutions employ multiple monocular cameras from different views to record two-dimensional (2D) projections of 3D vein vessels, which causes the following problems: 1) 2D images collected from limited views (two or three views) are insufficient for robust 3D vein vessel feature representation. Furthermore, image sequences of the same finger acquired from different views usually show significant differences. As a result, the existing works are still sensitive to positional variations of the fingers, specifically those caused by finger roll movements. 2) Using multiple cameras can lead to increased costs. Moreover, it is impossible to employ several cameras to acquire full-view images because of the limited space on capturing devices. To address the above issues, we present FV-LT, a Full-View Finger-Vein identification system based on a Local attention Transformer, by implementing an image acquisition device with a single camera. First, we design and implement a finger-vein acquisition prototype device that utilizes a single camera and a LED group to rotate along a finger for full-view image collection. This allows capturing all vein patterns concealed beneath human skin to form a complete representation of finger features. Second, given the full-view vein images, we propose a local attention transformer-based approach to extract dependency features of a token (a patch or an image) on its neighborhood’s tokens among image patches and among full-view images, respectively. These dependency features are shown to be robust to positional variations induced by finger rolls. Based on the public database of full-view finger-vein images captured by our designed device, we verify the performance of the proposed FV-LT. The experimental results show that FV-LT significantly outperforms existing 2D/multi-view based approaches with respect to improving the tolerance against finger roll and achieving the state-of-the-art identification accuracy.

A Vision Transformer-based Approach for Automatic COVID-19 Diagnosis on Chest X-ray Images

Son yıllarda ortaya çıkan yeni tip Koronavirüs hastalığı (COVID-19), dünya çapında sağlığı tehdit eden ciddi bir hastalık olmuştur. COVID-19 çok hızlı bir şekilde bulaşabilen ve ciddi ölüm artışları ile birçok endişeye zemin hazırlamıştır. Salgının evrensel boyuta taşınmasıyla bu hastalığın erken teşhisine yönelik birçok çalışma yapılmıştır. Erken teşhis ile hem ölümcül vakaların önüne geçilmiş olunmakta hem de salgının planlanması daha kolay olabilmektedir. X-ışını görüntülerinin zaman ve uygulanabilirlik açısından diğer görüntüleme tekniklerine nazaran çok daha avantajlı olması ve ayrıca ekonomik olması erken teşhis bazlı uygulama ve yöntemlerin bu görüntülerin üzerine yoğunlaşmasına neden olmuştur. Derin öğrenme yaklaşımları birçok hastalık teşhisinde olduğu gibi COVID-19 teşhisinde de çok büyük bir etki oluşturmuştur. Bu çalışmada, derin öğrenmenin CNN tabanlı yaklaşımları gibi daha önceki tekniklerinden ziyade en güncel ve çok daha popüler bir mimarisi olan transformatör yöntemine dayalı bir teşhis sistemi önerdik. Bu sistem, görü transformatör modelleri temelli bir yaklaşım ve yeni bir veri seti olan COVID-QU-Ex üzerinde COVID-19 hastalığının daha efektif bir teşhisini içermektedir. Deneysel çalışmalarda, görü transformatör modellerinin CNN modellerinden daha başarılı olduğu gözlemlenmiştir. Ayrıca, ViT-L16 modeli %96’nın üzerinde test doğruluğu ve F1-skoru sunarak, literatürde benzer çalışmalara kıyasla çok daha yüksek bir başarım göstermiştir.

BERT2OME: Prediction of 2'-O-Methylation Modifications From RNA Sequence by Transformer Architecture Based on BERT.

Recent work on language models has resulted in state-of-the-art performance on various language tasks. Among these, Bidirectional Encoder Representations from Transformers (BERT) has focused on contextualizing word embeddings to extract context and semantics of the words. On the other hand, post-transcriptional 2'-O-methylation (Nm) RNA modification is important in various cellular tasks and related to a number of diseases. The existing high-throughput experimental techniques take longer time to detect these modifications, and costly in exploring these functional processes. Here, to deeply understand the associated biological processes faster, we come up with an efficient method Bert2Ome to infer 2'-O-methylation RNA modification sites from RNA sequences. Bert2Ome combines BERT-based model with convolutional neural networks (CNN) to infer the relationship between the modification sites and RNA sequence content. Unlike the methods proposed so far, Bert2Ome assumes each given RNA sequence as a text and focuses on improving the modification prediction performance by integrating the pretrained deep learning-based language model BERT. Additionally, our transformer-based approach could infer modification sites across multiple species. According to 5-fold cross-validation, human and mouse accuracies were 99.15% and 94.35% respectively. Similarly, ROC AUC scores were 0.99, 0.94 for the same species. Detailed results show that Bert2Ome reduces the time consumed in biological experiments and outperforms the existing approaches across different datasets and species over multiple metrics. Additionally, deep learning approaches such as 2D CNNs are more promising in learning BERT attributes than more conventional machine learning methods.

CovSumm: an unsupervised transformer-cum-graph-based hybrid document summarization model for CORD-19.

The number of research articles published on COVID-19 has dramatically increased since the outbreak of the pandemic in November 2019. This absurd rate of productivity in research articles leads to information overload. It has increasingly become urgent for researchers and medical associations to stay up to date on the latest COVID-19 studies. To address information overload in COVID-19 scientific literature, the study presents a novel hybrid model named CovSumm, an unsupervised graph-based hybrid approach for single-document summarization, that is evaluated on the CORD-19 dataset. We have tested the proposed methodology on the scientific papers in the database dated from January 1, 2021 to December 31, 2021, consisting of 840 documents in total. The proposed text summarization is a hybrid of two distinctive extractive approaches (1) GenCompareSum (transformer-based approach) and (2) TextRank (graph-based approach). The sum of scores generated by both methods is used to rank the sentences for generating the summary. On the CORD-19, the recall-oriented understudy for gisting evaluation (ROUGE) score metric is used to compare the performance of the CovSumm model with various state-of-the-art techniques. The proposed method achieved the highest scores of ROUGE-1: 40.14%, ROUGE-2: 13.25%, and ROUGE-L: 36.32%. The proposed hybrid approach shows improved performance on the CORD-19 dataset when compared to existing unsupervised text summarization methods.