Pre-trained Transformer-based Models Research Articles

Unveiling Cyber Threat Actors: A Hybrid Deep Learning Approach for Behavior-based Attribution

In this paper, we leverage natural language processing and machine learning algorithms to profile threat actors based on their behavioral signatures to establish identification for soft attribution. Our unique dataset comprises various actors and the commands they have executed, with a significant proportion using the Cobalt Strike framework in August 2020-October 2022. We implemented a hybrid deep learning structure combining transformers and convolutional neural networks to benefit global and local contextual information within the sequence of commands, which provides a detailed view of the behavioral patterns of threat actors. We evaluated our hybrid architecture against pre-trained transformer-based models such as BERT, RoBERTa, SecureBERT, and DarkBERT with our high-count, medium-count, and low-count datasets. Hybrid architecture has achieved F1-score of 95.11% and an accuracy score of 95.13% on the high-count dataset, F1-score of 93.60% and accuracy score of 93.77% on the medium-count dataset, and F1-score of 88.95% and accuracy score of 89.25% on the low-count dataset. Our approach has the potential to substantially reduce the workload of incident response experts who are processing the collected cybersecurity data to identify patterns.

An in-depth evaluation of federated learning on biomedical natural language processing for information extraction

Language models (LMs) such as BERT and GPT have revolutionized natural language processing (NLP). However, the medical field faces challenges in training LMs due to limited data access and privacy constraints imposed by regulations like the Health Insurance Portability and Accountability Act (HIPPA) and the General Data Protection Regulation (GDPR). Federated learning (FL) offers a decentralized solution that enables collaborative learning while ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks encompassing 8 corpora using 6 LMs. Our results show that: (1) FL models consistently outperformed models trained on individual clients’ data and sometimes performed comparably with models trained with polled data; (2) with the fixed number of total data, FL models training with more clients produced inferior performance but pre-trained transformer-based models exhibited great resilience. (3) FL models significantly outperformed pre-trained LLMs with few-shot prompting.

Towards Data- and Compute-Efficient Fake-News Detection: An Approach Combining Active Learning and Pre-Trained Language Models

In today’s digital era, dominated by social media platforms such as Twitter, Facebook, and Instagram, the swift dissemination of misinformation represents a significant concern, impacting public sentiment and influencing pivotal global events. Promptly detecting such deceptive content with the help of Machine Learning models is crucial, yet it comes with the challenge of dealing with labelled examples for training these models. Impressive performance results were recently achieved by high-capacity pre-trained transformer-based models (e.g., BERT). Still, such models are too data- and compute-demanding for many critical application contexts where memory, time, and energy consumption must be limited. Here, we propose an innovative semi-supervised method for efficient and effective fake news detection using a content-oriented classifier based on a small-sized BERT embedder. After fine-tuning this model on the sole few labelled data available, an iterative Active Learning (AL) process is carried out, which benefits from limited experts’ feedback to acquire more labelled data for improving the model. The proposed method ensures good detection performances using a few training samples, reasonably small human intervention, and compute/memory costs.

Calibration of Transformer-Based Models for Identifying Stress and Depression in Social Media

In today’s fast-paced world, the rates of stress and depression present a surge. People use social media for expressing their thoughts and feelings through posts. Therefore, social media provide assistance for the early detection of mental health conditions. Existing methods mainly introduce feature extraction approaches and train shallow machine learning (ML) classifiers. For addressing the need of creating a large feature set and obtaining better performance, other research studies use deep neural networks or language models based on transformers. Despite the fact that transformer-based models achieve noticeable improvements, they cannot often capture rich factual knowledge. Although there have been proposed a number of studies aiming to enhance the pretrained transformer-based models with extra information or additional modalities, no prior work has exploited these modifications for detecting stress and depression through social media. In addition, although the reliability of a machine learning (ML) model’s confidence in its predictions is critical for high-risk applications, there is no prior work taken into consideration the model calibration. To resolve the above issues, we present the first study in the task of depression and stress detection in social media, which injects extra-linguistic information in transformer-based models, namely, bidirectional encoder representations from transformers (BERT) and MentalBERT. Specifically, the proposed approach employs a multimodal adaptation gate for creating the combined embeddings, which are given as input to a BERT (or MentalBERT) model. For taking into account the model calibration, we apply label smoothing. We test our proposed approaches in three publicly available datasets and demonstrate that the integration of linguistic features into transformer-based models presents a surge in performance. Also, the usage of label smoothing contributes to both the improvement of the model’s performance and the calibration of the model. We finally perform a linguistic analysis of the posts and show differences in language between stressful and nonstressful texts, as well as depressive and nondepressive posts.

Span Graph Transformer for Document-Level Named Entity Recognition

Named Entity Recognition (NER), which aims to identify the span and category of entities within text, is a fundamental task in natural language processing. Recent NER approaches have featured pre-trained transformer-based models (e.g., BERT) as a crucial encoding component to achieve state-of-the-art performance. However, due to the length limit for input text, these models typically consider text at the sentence-level and cannot capture the long-range contextual dependency within a document. To address this issue, we propose a novel Span Graph Transformer (SGT) method for document-level NER, which constructs long-range contextual dependencies at both the token and span levels. Specifically, we first retrieve relevant contextual sentences in the document for each target sentence, and jointly encode them by BERT to capture token-level dependencies. Then, our proposed model extracts candidate spans from each sentence and integrates these spans into a document-level span graph, where nested spans within sentences and identical spans across sentences are connected. By leveraging the power of Graph Transformer and well-designed position encoding, our span graph can fully exploit span-level dependencies within the document. Extensive experiments on both resource-rich nested and flat NER datasets, as well as low-resource distantly supervised NER datasets, demonstrate that proposed SGT model achieves better performance than previous state-of-the-art models.

Pre-Trained Transformer-Based Models for Text Classification Using Low-Resourced Ewe Language

Despite a few attempts to automatically crawl Ewe text from online news portals and magazines, the African Ewe language remains underdeveloped despite its rich morphology and complex "unique" structure. This is due to the poor quality, unbalanced, and religious-based nature of the crawled Ewe texts, thus making it challenging to preprocess and perform any NLP task with current transformer-based language models. In this study, we present a well-preprocessed Ewe dataset for low-resource text classification to the research community. Additionally, we have developed an Ewe-based word embedding to leverage the low-resource semantic representation. Finally, we have fine-tuned seven transformer-based models, namely BERT-based (cased and uncased), DistilBERT-based (cased and uncased), RoBERTa, DistilRoBERTa, and DeBERTa, using the preprocessed Ewe dataset that we have proposed. Extensive experiments indicate that the fine-tuned BERT-base-cased model outperforms all baseline models with an accuracy of 0.972, precision of 0.969, recall of 0.970, loss score of 0.021, and an F1-score of 0.970. This performance demonstrates the model’s ability to comprehend the low-resourced Ewe semantic representation compared to all other models, thus setting the fine-tuned BERT-based model as the benchmark for the proposed Ewe dataset.

Optimizing fake news detection for Arabic context: A multitask learning approach with transformers and an enhanced Nutcracker Optimization Algorithm

The rapid proliferation of news and posts across social media platforms has spawned a concerning wave of misinformation. Disseminating false information and news significantly threatens public health and safety. To address this critical issue, we present an innovative disinformation detection framework, leveraging the power of multi-task learning (MTL) and meta-heuristic techniques. Our framework harnesses the potential of an MTL approach and a state-of-the-art pre-trained Transformer-based model, enabling the extraction of comprehensive contextual features from Arabic social media posts. These contextual are input to an advanced feature selection (FS) model utilizing a modified Nutcracker Optimization Algorithm. Through extensive evaluation of diverse datasets of Arabic social media posts, our proposed framework achieves remarkable results. Notably, our framework attains an accuracy rate of 87% and 69% for binary and multi-classification, respectively. In addition, the developed method outperforms all compared algorithms. Our findings demonstrate the potency of our disinformation detection framework, serving as a robust tool in the battle against misinformation spread. By shedding light on the truth amidst the vast social media content, we can safeguard public health and empower individuals with reliable information.

Tracking pregnant women's mental health through social media: an analysis of reddit posts.

Present an artificial intelligence-enabled pipeline for estimating the prevalence of depression and general anxiety among pregnant women using texts from their social media posts. Use said pipeline to analyze mental health trends on subreddits frequented by pregnant women and report on interesting insights that could be helpful for policy-makers, clinicians, etc. We used pretrained transformer-based models to build a natural language processing pipeline that can automatically detect depressed pregnant women on social media and carry out topic modeling to detect their concerns. We detected depressed posts by pregnant women on Reddit and validated the performance of the depression classification model by carrying out topic modeling to reveal that depressive topics were detected. The proportion of potentially depressed surprisingly reduced during the pandemic (2020 and 2021). Queries related to antidepressants, such as Zoloft, and potential ways of managing mental health dominated discourse before the pandemic (2018 and 2019), whereas queries about pelvic pain and associated stress dominated the discourse during the pandemic. Supportive online communities could be a factor in alleviating stress related to the pandemic, hence the reduction in the proportion of depressed users during the pandemic. Stress during the pandemic has been associated with pelvic pain among pregnant women, and this trend is confirmed through topic modeling of depressive posts during the pandemic.

Comparison of pretrained transformer-based models for influenza and COVID-19 detection using social media text data in Saskatchewan, Canada.

The use of social media data provides an opportunity to complement traditional influenza and COVID-19 surveillance methods for the detection and control of outbreaks and informing public health interventions. The first aim of this study is to investigate the degree to which Twitter users disclose health experiences related to influenza and COVID-19 that could be indicative of recent plausible influenza cases or symptomatic COVID-19 infections. Second, we seek to use the Twitter datasets to train and evaluate the classification performance of Bidirectional Encoder Representations from Transformers (BERT) and variant language models in the context of influenza and COVID-19 infection detection. We constructed two Twitter datasets using a keyword-based filtering approach on English-language tweets collected from December 2016 to December 2022 in Saskatchewan, Canada. The influenza-related dataset comprised tweets filtered with influenza-related keywords from December 13, 2016, to March 17, 2018, while the COVID-19 dataset comprised tweets filtered with COVID-19 symptom-related keywords from January 1, 2020, to June 22, 2021. The Twitter datasets were cleaned, and each tweet was annotated by at least two annotators as to whether it suggested recent plausible influenza cases or symptomatic COVID-19 cases. We then assessed the classification performance of pre-trained transformer-based language models, including BERT-base, BERT-large, RoBERTa-base, RoBERT-large, BERTweet-base, BERTweet-covid-base, BERTweet-large, and COVID-Twitter-BERT (CT-BERT) models, on each dataset. To address the notable class imbalance, we experimented with both oversampling and undersampling methods. The influenza dataset had 1129 out of 6444 (17.5%) tweets annotated as suggesting recent plausible influenza cases. The COVID-19 dataset had 924 out of 11939 (7.7%) tweets annotated as inferring recent plausible COVID-19 cases. When compared against other language models on the COVID-19 dataset, CT-BERT performed the best, supporting the highest scores for recall (94.8%), F1(94.4%), and accuracy (94.6%). For the influenza dataset, BERTweet models exhibited better performance. Our results also showed that applying data balancing techniques such as oversampling or undersampling method did not lead to improved model performance. Utilizing domain-specific language models for monitoring users' health experiences related to influenza and COVID-19 on social media shows improved classification performance and has the potential to supplement real-time disease surveillance.

A Transformer-Based Approach to Authorship Attribution in Classical Arabic Texts

Authorship attribution (AA) is a field of natural language processing that aims to attribute text to its author. Although the literature includes several studies on Arabic AA in general, applying AA to classical Arabic texts has not gained similar attention. This study focuses on investigating recent Arabic pretrained transformer-based models in a rarely studied domain with limited research contributions: the domain of Islamic law. We adopt an experimental approach to investigate AA. Because no dataset has been designed specifically for this task, we design and build our own dataset using Islamic law digital resources. We conduct several experiments on fine-tuning four Arabic pretrained transformer-based models: AraBERT, AraELECTRA, ARBERT, and MARBERT. Results of the experiments indicate that for the task of attributing a given text to its author, ARBERT and AraELECTRA outperform the other models with an accuracy of 96%. We conclude that pretrained transformer models, specifically ARBERT and AraELECTRA, fine-tuned using the Islamic legal dataset, show significant results in applying AA to Islamic legal texts.

A Convolutional Neural Network-based gradient boosting framework for prediction of the band gap of photo-active catalysts

A recent trend in chemical synthesis is photo-catalysis, which uses photo-active catalyst materials that are semiconductor materials. A well-known electronic property of semiconducting materials is the band gap. A photo-catalyst’s desired band gap range is between 1.5 eV and 6.2 eV. A rational design and synthesis of photo-active catalysts require knowledge of the band gap as an initial screening parameter. Herein, we propose an integrated deep learning-based framework to classify the photo-active catalysts and predict their band gap using compositional features. To this extent, we have utilized the dataset extracted from the “catalyst hub” site by web scraping with the help of a Python script. Extensive data cleaning and pre-processing are done to make input data amenable for training the models. Also, more valuable features are made using two methods: (a) one hot-encoding and (b) calculating the mean of the embeddings of catalysts computed by Mat2Vec, a pre-trained transformer-based model. With the help of this generated feature set, we have proposed a two-stage deep-learning framework for classification and regression tasks. In the first stage, a 2D-Convolutional Neural Net (CNN)-based classifier is used to classify whether a catalyst belongs to the photo-active catalyst class. After the first stage screening, in the second stage, we use a 1D-VGG-based gradient boosting framework to predict the band gap of the photo-active catalyst only using compositional features as inputs. 2D-CNN for the classification task has an accuracy of 0.903 and 0.886 for the train and test datasets, respectively. Further, the proposed integrated model that uses 1D-Convolutional layers of VGG followed by the XGBoostRegressor has a test R2 of 0.750, much higher than baseline models reported in the literature.

Towards COVID-19 fake news detection using transformer-based models

The COVID-19 pandemic has resulted in a surge of fake news, creating public health risks. However, developing an effective way to detect such news is challenging, especially when published news involves mixing true and false information. Detecting COVID-19 fake news has become a critical task in the field of natural language processing (NLP). This paper explores the effectiveness of several machine learning algorithms and fine-tuning pre-trained transformer-based models, including Bidirectional Encoder Representations from Transformers (BERT) and COVID-Twitter-BERT (CT-BERT), for COVID-19 fake news detection. We evaluate the performance of different downstream neural network structures, such as CNN and BiGRU layers, added on top of BERT and CT-BERT with frozen or unfrozen parameters. Our experiments on a real-world COVID-19 fake news dataset demonstrate that incorporating BiGRU on top of the CT-BERT model achieves outstanding performance, with a state-of-the-art F1 score of 98%. These results have significant implications for mitigating the spread of COVID-19 misinformation and highlight the potential of advanced machine learning models for fake news detection.

The infinity vaccine war: linguistic regularities and audience engagement of vaccine debate on Twitter

Purpose As public health professionals strive to promote vaccines for inoculation efforts, fervent anti-vaccination movements are marshaling against it. This study is motived by a need to better understand the online discussion around vaccination. The authors identified the sentiments, emotions and topics of pro- and anti-vaxxers’ tweets, investigated their change since the pandemic started and further examined the associations between these content features and audiences’ engagement.Design/methodology/approach Utilizing a snowball sampling method, data were collected from the Twitter accounts of 100 pro-vaxxers (266,680 tweets) and 100 anti-vaxxers (248,425 tweets). The authors are adopting a zero-shot machine learning algorithm with a pre-trained transformer-based model for sentiment analysis and structural topic modeling to extract the topics. And the authors use the hurdle negative binomial model to test the relationships among sentiment/emotion, topics and engagement.Findings In general, pro-vaxxers used more positive tones and more emotions of joy in their tweets, while anti-vaxxers utilized more negative terms. The cues of sadness predominantly encourage retweets across the pro- and anti-vaccine corpus, while tweets amplifying the emotion of surprise are more attention-grabbing and getting more likes. Topic modeling of tweets yields the top 15 topics for pro- and anti-vaxxers separately. Among the pro-vaxxers’ tweets, the topics of “Child protection” and “COVID-19 situation” are positively predicting audiences’ engagement. For anti-vaxxers, the topics of “Supporting Trump,” “Injured children,” “COVID-19 situation,” “Media propaganda” and “Community building” are more appealing to audiences.Originality/value This study utilizes social media data and a state-of-art machine learning algorithm to generate insights into the development of emotionally appealing content and effective vaccine promotion strategies while combating coronavirus disease 2019 and moving toward a global recovery.Peer reviewThe peer review history for this article is available at https://publons.com/publon/10.1108/OIR-03-2022-0186

Roman Urdu Hate Speech Detection Using Transformer-Based Model for Cyber Security Applications.

Social media applications, such as Twitter and Facebook, allow users to communicate and share their thoughts, status updates, opinions, photographs, and videos around the globe. Unfortunately, some people utilize these platforms to disseminate hate speech and abusive language. The growth of hate speech may result in hate crimes, cyber violence, and substantial harm to cyberspace, physical security, and social safety. As a result, hate speech detection is a critical issue for both cyberspace and physical society, necessitating the development of a robust application capable of detecting and combating it in real-time. Hate speech detection is a context-dependent problem that requires context-aware mechanisms for resolution. In this study, we employed a transformer-based model for Roman Urdu hate speech classification due to its ability to capture the text context. In addition, we developed the first Roman Urdu pre-trained BERT model, which we named BERT-RU. For this purpose, we exploited the capabilities of BERT by training it from scratch on the largest Roman Urdu dataset consisting of 173,714 text messages. Traditional and deep learning models were used as baseline models, including LSTM, BiLSTM, BiLSTM + Attention Layer, and CNN. We also investigated the concept of transfer learning by using pre-trained BERT embeddings in conjunction with deep learning models. The performance of each model was evaluated in terms of accuracy, precision, recall, and F-measure. The generalization of each model was evaluated on a cross-domain dataset. The experimental results revealed that the transformer-based model, when directly applied to the classification task of the Roman Urdu hate speech, outperformed traditional machine learning, deep learning models, and pre-trained transformer-based models in terms of accuracy, precision, recall, and F-measure, with scores of 96.70%, 97.25%, 96.74%, and 97.89%, respectively. In addition, the transformer-based model exhibited superior generalization on a cross-domain dataset.

A Hybrid Multitask Learning Framework with a Fire Hawk Optimizer for Arabic Fake News Detection

The exponential spread of news and posts related to the COVID-19 pandemic on social media platforms led to the emergence of the disinformation phenomenon. The phenomenon of spreading fake information and news creates significant concern for the public health and safety of the population. In this paper, we propose a disinformation detection framework based on multi-task learning (MTL) and meta-heuristic algorithms in the context of the COVID-19 pandemic. The developed framework uses an MTL and a pre-trained transformer-based model to learn and extract contextual feature representations from Arabic social media posts. The extracted contextual representations are fed to an alternative feature selection technique which depends on modified version of the Fire Hawk Optimizer. The proposed framework, which aims to improve the disinformation detection rate, was evaluated on several datasets of Arabic social media posts. The experimental results show that the proposed framework can achieve accuracy of 59%. It obtained, at best, precision, recall, and F-measure of 53%, 71%, and 53%, respectively, on all datasets; and it outperformed the other algorithms in all measures.

Arabic abstractive text summarization using RNN-based and transformer-based architectures

Recently, the Transformer model architecture and the pre-trained Transformer-based language models have shown impressive performance when used in solving both natural language understanding and text generation tasks. Nevertheless, there is little research done on using these models for text generation in Arabic. This research aims at leveraging and comparing the performance of different model architectures, including RNN-based and Transformer-based ones, and different pre-trained language models, including mBERT, AraBERT, AraGPT2, and AraT5 for Arabic abstractive summarization. We first built an Arabic summarization dataset of 84,764 high-quality text-summary pairs. To use mBERT and AraBERT in the context of text summarization, we employed a BERT2BERT-based encoder-decoder model where we initialized both the encoder and decoder with the respective model weights. The proposed models have been tested using ROUGE metrics and manual human evaluation. We also compared their performance on out-of-domain data. Our pre-trained Transformer-based models give a large improvement in performance with ∼79% less data. We found that AraT5 scores ∼3 ROUGE higher than a BERT2BERT-based model that is initialized with AraBERT, indicating that an encoder-decoder pre-trained Transformer is more suitable for summarizing Arabic text. Also, both of these two models perform better than AraGPT2 by a clear margin, which we found to produce summaries with high readability but with relatively lesser quality. On the other hand, we found that both AraT5 and AraGPT2 are better at summarizing out-of-domain text. We released our models and dataset publicly11https://huggingface.co/malmarjeh,.22https://data.mendeley.com/datasets/7kr75c9h24/1

Multi-task Active Learning for Pre-trained Transformer-based Models

Abstract Multi-task learning, in which several tasks are jointly learned by a single model, allows NLP models to share information from multiple annotations and may facilitate better predictions when the tasks are inter-related. This technique, however, requires annotating the same text with multiple annotation schemes, which may be costly and laborious. Active learning (AL) has been demonstrated to optimize annotation processes by iteratively selecting unlabeled examples whose annotation is most valuable for the NLP model. Yet, multi-task active learning (MT-AL) has not been applied to state-of-the-art pre-trained Transformer-based NLP models. This paper aims to close this gap. We explore various multi-task selection criteria in three realistic multi-task scenarios, reflecting different relations between the participating tasks, and demonstrate the effectiveness of multi-task compared to single-task selection. Our results suggest that MT-AL can be effectively used in order to minimize annotation efforts for multi-task NLP models.1

Improving N-Best Rescoring in Under-Resourced Code-Switched Speech Recognition Using Pretraining and Data Augmentation

In this study, we present improvements in N-best rescoring of code-switched speech achieved by n-gram augmentation as well as optimised pretraining of long short-term memory (LSTM) language models with larger corpora of out-of-domain monolingual text. Our investigation specifically considers the impact of the way in which multiple monolingual datasets are interleaved prior to being presented as input to a language model. In addition, we consider the application of large pretrained transformer-based architectures, and present the first investigation employing these models in English-Bantu code-switched speech recognition. Our experimental evaluation is performed on an under-resourced corpus of code-switched speech comprising four bilingual code-switched sub-corpora, each containing a Bantu language (isiZulu, isiXhosa, Sesotho, or Setswana) and English. We find in our experiments that, by combining n-gram augmentation with the optimised pretraining strategy, speech recognition errors are reduced for each individual bilingual pair by 3.51% absolute on average over the four corpora. Importantly, we find that even speech recognition at language boundaries improves by 1.14% even though the additional data is monolingual. Utilising the augmented n-grams for lattice generation, we then contrast these improvements with those achieved after fine-tuning pretrained transformer-based models such as distilled GPT-2 and M-BERT. We find that, even though these language models have not been trained on any of our target languages, they can improve speech recognition performance even in zero-shot settings. After fine-tuning on in-domain data, these large architectures offer further improvements, achieving a 4.45% absolute decrease in overall speech recognition errors and a 3.52% improvement over language boundaries. Finally, a combination of the optimised LSTM and fine-tuned BERT models achieves a further gain of 0.47% absolute on average for three of the four language pairs compared to M-BERT. We conclude that the careful optimisation of the pretraining strategy used for neural network language models can offer worthwhile improvements in speech recognition accuracy even at language switches, and that much larger state-of-the-art architectures such as GPT-2 and M-BERT promise even further gains.

Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classification.

Pretrained contextual language models proposed in the recent past have been reported to achieve state-of-the-art performances in many natural language processing (NLP) tasks, including those involving health-related social media data. We sought to evaluate the effectiveness of different pretrained transformer-based models for social media-based health-related text classification tasks. An additional objective was to explore and propose effective pretraining strategies to improve machine learning performance on such datasets and tasks. We benchmarked six transformer-based models that were pretrained with texts from different domains and sources—BERT, RoBERTa, BERTweet, TwitterBERT, BioClinical_BERT, and BioBERT—on 22 social media-based health-related text classification tasks. For the top-performing models, we explored the possibility of further boosting performance by comparing several pretraining strategies: domain-adaptive pretraining (DAPT), source-adaptive pretraining (SAPT), and a novel approach called topic specific pretraining (TSPT). We also attempted to interpret the impacts of distinct pretraining strategies by visualizing document-level embeddings at different stages of the training process. RoBERTa outperformed BERTweet on most tasks, and better than others. BERT, TwitterBERT, BioClinical_BERT and BioBERT consistently underperformed. For pretraining strategies, SAPT performed better or comparable to the off-the-shelf models, and significantly outperformed DAPT. SAPT + TSPT showed consistently high performance, with statistically significant improvement in three tasks. Our findings demonstrate that RoBERTa and BERTweet are excellent off-the-shelf models for health-related social media text classification, and extended pretraining using SAPT and TSPT can further improve performance.

ProtTrans-Glutar: Incorporating Features From Pre-trained Transformer-Based Models for Predicting Glutarylation Sites.

Lysine glutarylation is a post-translational modification (PTM) that plays a regulatory role in various physiological and biological processes. Identifying glutarylated peptides using proteomic techniques is expensive and time-consuming. Therefore, developing computational models and predictors can prove useful for rapid identification of glutarylation. In this study, we propose a model called ProtTrans-Glutar to classify a protein sequence into positive or negative glutarylation site by combining traditional sequence-based features with features derived from a pre-trained transformer-based protein model. The features of the model were constructed by combining several feature sets, namely the distribution feature (from composition/transition/distribution encoding), enhanced amino acid composition (EAAC), and features derived from the ProtT5-XL-UniRef50 model. Combined with random under-sampling and XGBoost classification method, our model obtained recall, specificity, and AUC scores of 0.7864, 0.6286, and 0.7075 respectively on an independent test set. The recall and AUC scores were notably higher than those of the previous glutarylation prediction models using the same dataset. This high recall score suggests that our method has the potential to identify new glutarylation sites and facilitate further research on the glutarylation process.