Transformer-based Models Research Articles

Task-Specific Transformer-Based Language Models in Health Care: Scoping Review

Background Transformer-based language models have shown great potential to revolutionize health care by advancing clinical decision support, patient interaction, and disease prediction. However, despite their rapid development, the implementation of transformer-based language models in health care settings remains limited. This is partly due to the lack of a comprehensive review, which hinders a systematic understanding of their applications and limitations. Without clear guidelines and consolidated information, both researchers and physicians face difficulties in using these models effectively, resulting in inefficient research efforts and slow integration into clinical workflows. Objective This scoping review addresses this gap by examining studies on medical transformer-based language models and categorizing them into 6 tasks: dialogue generation, question answering, summarization, text classification, sentiment analysis, and named entity recognition. Methods We conducted a scoping review following the Cochrane scoping review protocol. A comprehensive literature search was performed across databases, including Google Scholar and PubMed, covering publications from January 2017 to September 2024. Studies involving transformer-derived models in medical tasks were included. Data were categorized into 6 key tasks. Results Our key findings revealed both advancements and critical challenges in applying transformer-based models to health care tasks. For example, models like MedPIR involving dialogue generation show promise but face privacy and ethical concerns, while question-answering models like BioBERT improve accuracy but struggle with the complexity of medical terminology. The BioBERTSum summarization model aids clinicians by condensing medical texts but needs better handling of long sequences. Conclusions This review attempted to provide a consolidated understanding of the role of transformer-based language models in health care and to guide future research directions. By addressing current challenges and exploring the potential for real-world applications, we envision significant improvements in health care informatics. Addressing the identified challenges and implementing proposed solutions can enable transformer-based language models to significantly improve health care delivery and patient outcomes. Our review provides valuable insights for future research and practical applications, setting the stage for transformative advancements in medical informatics.

Transfer Learning with Transformer-Based Models for Mine Water Inrush Prediction: A Multivariate Analysis Using Sparse and Imbalanced Monitoring Data

Transfer Learning with Transformer-Based Models for Mine Water Inrush Prediction: A Multivariate Analysis Using Sparse and Imbalanced Monitoring Data

Enhancing misogyny detection in bilingual texts using explainable AI and multilingual fine-tuned transformers

AbstractGendered disinformation undermines women’s rights, democratic principles, and national security by worsening societal divisions through authoritarian regimes’ intentional weaponization of social media. Online misogyny represents a harmful societal issue, threatening to transform digital platforms into environments that are hostile and inhospitable to women. Despite the severity of this issue, efforts to persuade digital platforms to strengthen their protections against gendered disinformation are frequently ignored, highlighting the difficult task of countering online misogyny in the face of commercial interests. This growing concern underscores the need for effective measures to create safer online spaces, where respect and equality prevail, ensuring that women can participate fully and freely without the fear of harassment or discrimination. This study addresses the challenge of detecting misogynous content in bilingual (English and Italian) online communications. Utilizing FastText word embeddings and explainable artificial intelligence techniques, we introduce a model that enhances both the interpretability and accuracy in detecting misogynistic language. To conduct an in-depth analysis, we implemented a range of experiments encompassing classic machine learning methodologies and conventional deep learning approaches to the recent transformer-based models incorporating both language-specific and multilingual capabilities. This paper enhances the methodologies for detecting misogyny by incorporating incremental learning for cutting-edge datasets containing tweets and posts from different sources like Facebook, Twitter, and Reddit, with our proposed approach outperforming these datasets in metrics such as accuracy, F1-score, precision, and recall. This process involved refining hyperparameters, employing optimization techniques, and utilizing generative configurations. By implementing Local Interpretable Model-agnostic Explanations (LIME), we further elucidate the rationale behind the model’s predictions, enhancing understanding of its decision-making process.

Image forgery localization integrating multi-scale and boundary features

Abstract Image forgery localization identifies tampered regions within an image by extracting distinctive forgery features. Current methods mainly use convolutional neural networks (CNNs) to extract features. However, CNNs’ limited receptive field emphasizes local features, impeding the global modeling of crucial lower-level features like edges and textures, leading to decreased precision. Moreover, prior methods use pyramid networks for multi-scale feature extraction but show deficiencies in multi-scale and interlayer modeling, leading to inadequate multi-scale information representation and limiting flexibility to tampered regions of varying sizes. To address these issues, this paper proposes a Transformer-based model integrating multi-scale and boundary features. The model employs a Pyramid Vision Transformer as the encoder, using self-attention over convolution to enhance global context modeling. Building on this, the model incorporates a multi-scale feature enhancement module that enriches forgery features by paralleling various convolutional layers. Features at various encoder stages are integrated through a cross-stage interaction module, enabling multi-level feature correlation for a strong feature representation. Furthermore, the model includes a forgery boundary information-guided branch, which focuses precisely on tampered region structures without introducing irrelevant noise. Experiments demonstrate that our model surpasses previous methods in localization accuracy, with F1 and AUC improving by 8.5% and 2.2% in pre-training, respectively.

Explainable Pre-Trained Language Models for Sentiment Analysis in Low-Resourced Languages

Sentiment analysis is a crucial tool for measuring public opinion and understanding human communication across digital social media platforms. However, due to linguistic complexities and limited data or computational resources, it is under-represented in many African languages. While state-of-the-art Afrocentric pre-trained language models (PLMs) have been developed for various natural language processing (NLP) tasks, their applications in eXplainable Artificial Intelligence (XAI) remain largely unexplored. In this study, we propose a novel approach that combines Afrocentric PLMs with XAI techniques for sentiment analysis. We demonstrate the effectiveness of incorporating attention mechanisms and visualization techniques in improving the transparency, trustworthiness, and decision-making capabilities of transformer-based models when making sentiment predictions. To validate our approach, we employ the SAfriSenti corpus, a multilingual sentiment dataset for South African under-resourced languages, and perform a series of sentiment analysis experiments. These experiments enable comprehensive evaluations, comparing the performance of Afrocentric models against mainstream PLMs. Our results show that the Afro-XLMR model outperforms all other models, achieving an average F1-score of 71.04% across five tested languages, and the lowest error rate among the evaluated models. Additionally, we enhance the interpretability and explainability of the Afro-XLMR model using Local Interpretable Model-Agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP). These XAI techniques ensure that sentiment predictions are not only accurate and interpretable but also understandable, fostering trust and reliability in AI-driven NLP technologies, particularly in the context of African languages.

Improving ranking-based question answering with weak supervision for low-resource Qur’anic texts

AbstractThis work tackles the challenge of ranking-based machine reading comprehension (MRC), where a question answering (QA) system generates a ranked list of relevant answers for each question instead of simply extracting a single answer. We highlight the limitations of traditional learning methods in this setting, particularly under limited training data. To address these issues, we propose a novel ranking-inspired learning method that focuses on ranking multiple answer spans instead of single answer extraction. This method leverages lexical overlap as weak supervision to guide the ranking process. We evaluate our approach on the Qur’an Reading Comprehension Dataset (QRCD), a low-resource Arabic dataset over the Holy Qur’an. We employ transfer learning with external resources to fine-tune various transformer-based models, mitigating the low-resource challenge. Experimental results demonstrate that our proposed method significantly outperforms standard mechanisms across different models. Furthermore, we show its better alignment with the ranking-based MRC task and the effectiveness of external resources for this low-resource dataset. Our best performing model achieves a state-of-the-art partial Reciprocal Rank (pRR) score of 63.82%, surpassing the previous best-known score of 58.60%. To foster further research, we release code [GitHub repository:github.com/mohammed-elkomy/weakly-supervised-mrc], trained models, and detailed experiments to the community.

DeePathNet: a transformer-based deep learning model integrating multi-omic data with cancer pathways.

Multi-omic data analysis incorporating machine learning has the potential to significantly improve cancer diagnosis and prognosis. Traditional machine learning methods are usually limited to omic measurements, omitting existing domain knowledge, such as the biological networks that link molecular entities in various omic data types. Here we develop a Transformer-based explainable deep learning model, DeePathNet, which integrates cancer-specific pathway information into multi-omic data analysis. Using a variety of big datasets, including ProCan-DepMapSanger, CCLE, and TCGA, we demonstrate and validate that DeePathNet outperforms traditional methods for predicting drug response and classifying cancer type and subtype. Combining biomedical knowledge and state-of-the-art deep learning methods, DeePathNet enables biomarker discovery at the pathway level, maximizing the power of data-driven approaches to cancer research. DeePathNet is available on GitHub at https://github.com/CMRI-ProCan/DeePathNet.

Smart Caption: Intelligent Image Description Using Transformer Decoder

The need for automated image description systems has grown significantly with the rise of multimedia content across diverse domains such as social media, digital libraries, and accessibility technologies. Smart Caption presents a novel approach for generating intelligent and context-aware image descriptions by utilizing a combination of Vision Transformers (ViTs) and Natural Language Processing (NLP) Transformer decoders. Unlike traditional convolution-based methods, Vision Transformers treat images as sequences of patches, enabling them to capture global image features more effectively. These extracted visual features are then processed by a Transformer-based decoder to produce coherent and contextually appropriate captions, bridging the gap between image recognition and natural language generation. Our approach focuses on leveraging the attention mechanisms inherent in both Vision and NLP Transformers to enhance the quality of image descriptions. The ViT architecture is designed to focus on relevant regions within an image, while the NLP Transformer decoder is adept at generating fluent and detailed descriptions by attending to the most significant visual features. This two-stage process improves the precision of object detection, relationship understanding, and scene context in generated captions. Experimental results demonstrate that Smart Caption out performs traditional methods in terms of accuracy, contextual relevance, and fluency, marking a significant step forward in the field of automated image captioning. Through this research, we aim to provide a scalable, efficient, and intelligent solution for image description, with potential applications in areas such as accessibility tools for visually impaired users, digital content indexing, and automated content generation. Our findings highlight the strengths of transformer-based models in bridging vision and language tasks, offering promising directions for future research in multimodal AI..

Deep Learning-Based Classification of Macrofungi: Comparative Analysis of Advanced Models for Accurate Fungi Identification

This study focuses on the classification of six different macrofungi species using advanced deep learning techniques. Fungi species, such as Amanita pantherina, Boletus edulis, Cantharellus cibarius, Lactarius deliciosus, Pleurotus ostreatus and Tricholoma terreum were chosen based on their ecological importance and distinct morphological characteristics. The research employed 5 different machine learning techniques and 12 deep learning models, including DenseNet121, MobileNetV2, ConvNeXt, EfficientNet, and swin transformers, to evaluate their performance in identifying fungi from images. The DenseNet121 model demonstrated the highest accuracy (92%) and AUC score (95%), making it the most effective in distinguishing between species. The study also revealed that transformer-based models, particularly the swin transformer, were less effective, suggesting room for improvement in their application to this task. Further advancements in macrofungi classification could be achieved by expanding datasets, incorporating additional data types such as biochemical, electron microscopy, and RNA/DNA sequences, and using ensemble methods to enhance model performance. The findings contribute valuable insights into both the use of deep learning for biodiversity research and the ecological conservation of macrofungi species.

Advancements in Image Recognition: Comparing CNNs and Vision Transformers

Abstract. This paper explores advancements in image recognition technologies, highlighting the shift from conventional methodologies to contemporary deep learning techniques, specifically focusing on Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The study examines key architectures including CNNs, and various Transformer-based models, analyzing their performance evaluating their effectiveness in diverse tasks such as image classification, object detection, and facial recognition. The research highlights the strengths and limitations of CNNs and ViTs, focusing on their ability to handle complex and diverse datasets. A detailed comparative analysis is conducted, emphasizing performance metrics, robustness, and adaptability across different image recognition scenarios. The results reveal that while CNNs excel in traditional image processing tasks, ViTs demonstrate significant improvements in capturing long-range dependencies, thereby enhancing recognition accuracy in more complex contexts. This analysis offers critical perspectives on selecting and applying image recognition models, guiding future exploration and practical use in various industries. It underscores the impact of deep learning innovations on advancing image recognition capabilities and highlights potential directions for ongoing development in the field.

Cone-Beam CT to CT Image Translation Using a Transformer-Based Deep Learning Model for Prostate Cancer Adaptive Radiotherapy.

Cone-beam computed tomography (CBCT) is widely utilized in image-guided radiation therapy; however, its image quality is poor compared to planning CT (pCT), thus restricting its utility for adaptive radiotherapy (ART). Our objective was to enhance CBCT image quality utilizing a transformer-based deep learning model, SwinUNETR, which we compared with a conventional convolutional neural network (CNN) model, U-net. This retrospective study involved 260 patients undergoing prostate radiotherapy, with 245 patients used for training and 15 patients reserved as an independent hold-out test dataset. Employing a CycleGAN framework, we generated synthetic CT (sCT) images from CBCT images, employing SwinUNETR and U-net as generators. We evaluated sCT image quality and assessed its dosimetric impact for photon therapy through gamma analysis and dose-volume histogram (DVH) comparisons. The mean absolute error values for the CT numbers, calculated using all voxels within the patient's body contour and taking the pCT images as a reference, were 84.07, 73.49, and 64.69 Hounsfield units for CBCT, U-net, and SwinUNETR images, respectively. Gamma analysis revealed superior agreement between the dose on the pCT images and on the SwinUNETR-based sCT plans compared to those based on U-net. DVH parameters calculated on the SwinUNETR-based sCT deviated by < 1% from those in pCT plans. Our study showed that, compared to the U-net model, SwinUNETR could proficiently generate more precise sCT images from CBCT images, facilitating more accurate dose calculations. This study demonstrates the superiority of transformer-based models over conventional CNN-based approaches for CBCT-to-CT translation, contributing to the advancement of image synthesis techniques in ART.

A short trajectory is all you need: A transformer-based model for long-time dissipative quantum dynamics.

In this Communication, we demonstrate that a deep artificial neural network based on a transformer architecture with self-attention layers can predict the long-time population dynamics of a quantum system coupled to a dissipative environment provided that the short-time population dynamics of the system is known. The transformer neural network model developed in this work predicts the long-time dynamics of spin-boson model efficiently and very accurately across different regimes, from weak system-bath coupling to strong coupling non-Markovian regimes. Our model is more accurate than classical forecasting models, such as recurrent neural networks, and is comparable to the state-of-the-art models for simulating the dynamics of quantum dissipative systems based on kernel ridge regression.

A transformer-based deep learning model for identifying the occurrence of acute hematogenous osteomyelitis and predicting blood culture results

BackgroundAcute hematogenous osteomyelitis is the most common form of osteomyelitis in children. In recent years, the incidence of osteomyelitis has been steadily increasing. For pediatric patients, clearly describing their symptoms can be quite challenging, which often necessitates the use of complex diagnostic methods, such as radiology. For those who have been diagnosed, the ability to culture the pathogenic bacteria significantly affects their treatment plan.MethodA total of 634 patients under the age of 18 were included, and the correlation between laboratory indicators and osteomyelitis, as well as several diagnoses often confused with osteomyelitis, was analyzed. Based on this, a Transformer-based deep learning model was developed to identify osteomyelitis patients. Subsequently, the correlation between laboratory indicators and the length of hospital stay for osteomyelitis patients was examined. Finally, the correlation between the successful cultivation of pathogenic bacteria and laboratory indicators in osteomyelitis patients was analyzed, and a deep learning model was established for prediction.ResultThe laboratory indicators of patients are correlated with the presence of acute hematogenous osteomyelitis, and the deep learning model developed based on this correlation can effectively identify patients with acute hematogenous osteomyelitis. The laboratory indicators of patients with acute hematogenous osteomyelitis can partially reflect their length of hospital stay. Although most laboratory indicators lack a direct correlation with the ability to culture pathogenic bacteria in patients with acute hematogenous osteomyelitis, our model can still predict whether the bacteria can be successfully cultured.ConclusionLaboratory indicators, as easily accessible medical information, can identify osteomyelitis in pediatric patients. They can also predict whether pathogenic bacteria can be successfully cultured, regardless of whether the patient has received antibiotics beforehand. This not only simplifies the diagnostic process for pediatricians but also provides a basis for deciding whether to use empirical antibiotic therapy or discontinue treatment for blood cultures.

BreakNet: discontinuity-resilient multi-scale transformer segmentation of retinal layers

Visible light optical coherence tomography (vis-OCT) is gaining traction for retinal imaging due to its high resolution and functional capabilities. However, the significant absorption of hemoglobin in the visible light range leads to pronounced shadow artifacts from retinal blood vessels, posing challenges for accurate layer segmentation. In this study, we present BreakNet, a multi-scale Transformer-based segmentation model designed to address boundary discontinuities caused by these shadow artifacts. BreakNet utilizes hierarchical Transformer and convolutional blocks to extract multi-scale global and local feature maps, capturing essential contextual, textural, and edge characteristics. The model incorporates decoder blocks that expand pathways to enhance the extraction of fine details and semantic information, ensuring precise segmentation. Evaluated on rodent retinal images acquired with prototype vis-OCT, BreakNet demonstrated superior performance over state-of-the-art segmentation models, such as TCCT-BP and U-Net, even when faced with limited-quality ground truth data. Our findings indicate that BreakNet has the potential to significantly improve retinal quantification and analysis.

Exploring protein natural diversity in environmental microbiomes with DeepMetagenome.

Protein natural diversity offers a vast sequence space for protein engineering, and deep learning enables its detection from metagenomes/proteomes without prior assumptions. DeepMetagenome, a Python-based method, explores protein diversity through modules for training and analyzing sequence datasets. The deep learning model includes Embedding, Conv1D, LSTM, and Dense layers, with sequence feature analysis for data cleaning. Applied to metallothioneins from a database of over 146 million coding features, DeepMetagenome identified over 500 high-confidence metallothionein sequences, outperforming DIAMOND and CNN-based models. It showed stable performance compared to a Transformer-based model over 25 epochs. Among 23 synthesized sequences, 20 exhibited metal resistance. The tool also successfully explored the diversity of three additional protein families and is freely available on GitHub with detailed instructions.

Multi-Label Topic Classification in a Twitter Corpus of Public Communication of Science in Mexican Spanish

In the context of Mexico, comprehensive studies on the public communication of science (PCS) through social networks remain an unaddressed area of research. To address this gap, the present work is conducted from the perspective of natural language processing (NLP). The objective of this study is to develop and evaluate an automatic multilabel topic classification system for PCS tweets published in Mexico. This is achieved by training various machine learning models, which include traditional algorithms and transformer-based models. Utilizing a manually labeled corpus that identifies eighteen distinct areas or themes of science, the study evaluates and compares several approaches for the automatic identification and classification of thematic areas within PCS tweets. The findings indicate that transformer-based models, such as XLM-RoBERTa, demonstrate superior performance compared to classic algorithms, while the emerging LLM models, such as BLOOM, present a promising alternative for a range of NLP tasks.

News dissemination: a semantic approach to barrier classification

AbstractThe dissemination of information worldwide is significantly facilitated by the news media, with many events having global relevance across various regions. However, certain news events receive limited coverage restricted to specific geographic areas, due to the barriers that hinder the spread of information. These barriers can be attributed to political, geographical, economic, cultural, or linguistic factors. In this research, we propose an approach for classifying these barriers by extracting semantic information from news articles using Wikipedia-concepts. Our methodology involves the collection of news articles, each annotated to indicate the specific barrier types, leveraging metadata from news publishers. Subsequently, we employ Wikipedia-concepts, in conjunction with the content of the news articles, as features to determine the barriers to news dissemination. Our approach is then compared with traditional text classification techniques, deep learning methods, and transformer-based models. We have performed experiments on news articles from ten categories of topics including health, sports, business, etc. The findings indicate that 1) Utilizing semantic knowledge yields distinct concepts across the ten categories, thereby enhancing the effectiveness and speed of the classification model. 2) The proposed approach, incorporating Wikipedia-concepts-based semantic knowledge, leads to improved performance in barrier classification when compared to using solely the body text of news articles. Specifically, there is an increase in the average F1-scores for four out of five barriers, with the economic barrier rising from 0.65 to 0.68, the linguistic barrier from 0.71 to 0.72, the political barrier from 0.68 to 0.70, and the geographical barrier from 0.63 to 0.68.

Ensemble based high performance deep learning models for fake news detection

Social media has emerged as a dominant platform where individuals freely share opinions and communicate globally. Its role in disseminating news worldwide is significant due to its easy accessibility. However, the increase in the use of these platforms presents severe risks for potentially misleading people. Our research aims to investigate different techniques within machine learning, deep learning, and ensemble learning frameworks in Arabic fake news detection. We integrated FastText word embeddings with various machine learning and deep learning methods. We then leveraged advanced transformer-based models, including BERT, XLNet, and RoBERTa, optimizing their performance through careful hyperparameter tuning. The research methodology involves utilizing two Arabic news article datasets, AFND and ARABICFAKETWEETS datasets, categorized into fake and real subsets and applying comprehensive preprocessing techniques to the text data. Four hybrid deep learning models are presented: CNN-LSTM, RNN-CNN, RNN-LSTM, and Bi-GRU-Bi-LSTM. The Bi-GRU-Bi-LSTM model demonstrated superior performance regarding the F1 score, accuracy, and loss metrics. The precision, recall, F1 score, and accuracy of the hybrid Bi-GRU-Bi-LSTM model on the AFND Dataset are 0.97, 0.97, 0.98, and 0.98, and on the ARABICFAKETWEETS dataset are 0.98, 0.98, 0.99, and 0.99 respectively. The study’s primary conclusion is that when spotting fake news in Arabic, the Bi-GRU-Bi-LSTM model outperforms other models by a significant margin. It significantly aids the global fight against false information by setting the stage for future research to expand fake news detection to multiple languages.

Uncovering suggestions in MOOC discussion forums: a transformer-based approach

The field of natural language processing has experienced significant advances in recent years, but these advances have not yet resulted in improved analytics for instructors on MOOC platforms. Valuable information, such as suggestions, is generated in the comment forums of these courses, but due to their volume, manual processing is often impractical. This study examines the feasibility of fine-tuning and effectively utilizing state-of-the-art deep learning models to identify comments that contain suggestions in MOOC forums. The main challenges encountered are the lack of labeled datasets from the MOOC context for fine-tuning classification models and the soaring computational cost of this training. For this study, we manually collected and labeled 2228 comments in Spanish and English from 5 MOOCs and scraped 1.4 million MOOC reviews from 3 platforms. We fine-tuned and evaluated 4 pretrained models based on the transformer architecture and 3 traditional machine learning models to compare their effectiveness in the suggestion mining task in this domain. Transformer-based models proved to be highly effective in this task/domain combination, achieving performance levels that matched or exceeded those deemed appropriate in other contexts and were significantly greater than those achieved by traditional models. Domain adaptation led to improved linguistic understanding of the target domain; however, in this project, this approach did not translate into an observable improvement in suggestion mining. The automated identification of comments that can be labeled as suggestions can result in considerable time savings for instructors, especially considering that less than a quarter of the analyzed comments contain suggestions.

Semantic-guided attention and adaptive gating for document-level relation extraction

In natural language processing, document-level relation extraction is a complex task that aims to predict the relationships among entities by capturing contextual interactions from an unstructured document. Existing graph- and transformer-based models capture long-range relational facts across sentences. However, they still cannot fully exploit the semantic information from multiple interactive sentences, resulting in the exclusion of influential sentences for related entities. To address this problem, a novel Semantic-guided Attention and Adaptively Gated (SAAG) model is developed for document-level relation extraction. First, a semantic-guided attention module is designed to guide sentence representation by assigning different attention scores to different words. The multihead attention mechanism is then used to capture the attention of different subspaces further to generate a document context representation. Finally, the SAAG model exploits the semantic information by leveraging a gating mechanism that can dynamically distinguish between local and global contexts. The experimental results demonstrate that the SAAG model outperforms previous models on two public datasets.