Language Modeling Approach Research Articles

A modular protein language modelling approach to immunogenicity prediction.

Neoantigen immunogenicity prediction is a highly challenging problem in the development of personalised medicines. Low reactivity rates in called neoantigens result in a difficult prediction scenario with limited training datasets. Here we describe ImmugenX, a modular protein language modelling approach to immunogenicity prediction for CD8+ reactive epitopes. ImmugenX comprises of a pMHC encoding module trained on three pMHC prediction tasks, an optional TCR encoding module and a set of context specific immunogenicity prediction head modules. Compared with state-of-the-art models for each task, ImmugenX's encoding module performs comparably or better on pMHC binding affinity, eluted ligand prediction and stability tasks. ImmugenX outperforms all compared models on pMHC immunogenicity prediction (Area under the receiver operating characteristic curve = 0.619, average precision: 0.514), with a 7% increase in average precision compared to the next best model. ImmugenX shows further improved performance on immunogenicity prediction with the integration of TCR context information. ImmugenX performance is further analysed for interpretability, which locates areas of weakness found across existing immunogenicity models and highlight possible biases in public datasets.

Language Modeling Screens Parkinson's Disease with Self-reported Questionnaires.

Parkinson's disease (PD) is a growing public health challenge associated with the aging population. Current diagnostic methods rely on motor symptoms and invasive procedures, making early detection difficult. This study established a transferable artificial intelligence (AI) model, Quest2Dx, to analyze health questionnaires to enable low-cost and non-invasive PD diagnosis. Quest2Dx tackles the common challenges of missing responses and required specific modeling for each questionnaire by developing a novel language modeling approach to allow the model transfer across different questionnaires and to enhance the interpretability. Evaluated on the PPMI and Fox Insight datasets, Quest2Dx achieved AUROCs of 0.977 and 0.974, respectively, significantly outperforming existing methods. Additionally, cross-questionnaire validation achieved AUROCs of 0.920 and 0.952, respectively, from PPMI to Fox Insight and vice versa. Quest2Dx also identified key predictors from the list of questions to provide further insights. The validated technology elucidates a promising path for PD screening in primary-care settings.

Extracting lung cancer staging descriptors from pathology reports: A generative language model approach

BackgroundIn oncology, electronic health records contain textual key information for the diagnosis, staging, and treatment planning of patients with cancer. However, text data processing requires a lot of time and effort, which limits the utilization of these data. Recent advances in natural language processing (NLP) technology, including large language models, can be applied to cancer research. Particularly, extracting the information required for the pathological stage from surgical pathology reports can be utilized to update cancer staging according to the latest cancer staging guidelines. ObjectivesThis study has two main objectives. The first objective is to evaluate the performance of extracting information from text-based surgical pathology reports and determining pathological stages based on the extracted information using fine-tuned generative language models (GLMs) for patients with lung cancer. The second objective is to determine the feasibility of utilizing relatively small GLMs for information extraction in a resource-constrained computing environment. MethodsLung cancer surgical pathology reports were collected from the Common Data Model database of Seoul National University Bundang Hospital (SNUBH), a tertiary hospital in Korea. We selected 42 descriptors necessary for tumor-node (TN) classification based on these reports and created a gold standard with validation by two clinical experts. The pathology reports and gold standard were used to generate prompt-response pairs for training and evaluating GLMs which then were used to extract information required for staging from pathology reports. ResultsWe evaluated the information extraction performance of six trained models as well as their performance in TN classification using the extracted information. The Deductive Mistral-7B model, which was pre-trained with the deductive dataset, showed the best performance overall, with an exact match ratio of 92.24% in the information extraction problem and an accuracy of 0.9876 (predicting T and N classification concurrently) in classification. ConclusionThis study demonstrated that training GLMs with deductive datasets can improve information extraction performance, and GLMs with a relatively small number of parameters at approximately seven billion can achieve high performance in this problem. The proposed GLM-based information extraction method is expected to be useful in clinical decision-making support, lung cancer staging and research.

Classifier Chains for LOINC Transcoding.

Mapping clinical observations and medical test results into the standardized vocabulary LOINC is a prerequisite for exchanging clinical data between health information systems and ensuring efficient interoperability. We present a comparison of three approaches for LOINC transcoding applied to French data collected from real-world settings. These approaches include both a state-of-the-art language model approach and a classifier chains approach. Our study demonstrates that we successfully improve the performance of the baselines using the classifier chains approach and compete effectively with state-of-the-art language models. Our approach proves to be efficient, cost-effective despite reproducibility challenges and potential for future optimizations and dataset testing.

Defect-scanner: a comparative empirical study on language model and deep learning approach for software vulnerability detection

Defect-scanner: a comparative empirical study on language model and deep learning approach for software vulnerability detection

Public Sentiment Towards Mandatory Halal Certification: A Large Language Model (LLM) Approach

Public Sentiment Towards Mandatory Halal Certification: A Large Language Model (LLM) Approach

Emotion AWARE: an artificial intelligence framework for adaptable, robust, explainable, and multi-granular emotion analysis

Emotions are fundamental to human behaviour. How we feel, individually and collectively, determines how humanity evolves and advances into our shared future. The rapid digitalisation of our personal, social and professional lives means we are frequently using digital media to express, understand and respond to emotions. Although recent developments in Artificial Intelligence (AI) are able to analyse sentiment and detect emotions, they are not effective at comprehending the complexity and ambiguity of digital emotion expressions in knowledge-focused activities of customers, people, and organizations. In this paper, we address this challenge by proposing a novel AI framework for the adaptable, robust, and explainable detection of multi-granular assembles of emotions. This framework consolidates lexicon generation and finetuned Large Language Model (LLM) approaches to formulate multi-granular assembles of two, eight and fourteen emotions. The framework is robust to ambiguous emotion expressions that are implied in conversation, adaptable to domain-specific emotion semantics, and the assembles are explainable using constituent terms and intensity. We conducted nine empirical studies using datasets representing diverse human emotion behaviours. The results of these studies comprehensively demonstrate and evaluate the core capabilities of the framework, and consistently outperforms state-of-the-art approaches in adaptable, robust, and explainable multi-granular emotion detection.

Managerial sentiment, life cycle and corporate investment: a large language model approach

Managerial sentiment, life cycle and corporate investment: a large language model approach

Integrating AI with MBSE for Data Extraction from Medical Standards

AbstractThe growing adoption of Model‐Based Systems Engineering (MBSE) in the medical sector has prompted a significant emphasis on the digitization of medical standards into norm models. This transformation promotes consistency and allows for tracing system model elements to the corresponding norm model elements. Despite these efforts, the current digitization activities heavily rely on manual extraction and transformation, particularly from PDF documents into SysML models. Concurrently, the proliferation of Artificial Intelligence (AI) applications in recent years presents an opportunity to automate such activities. This paper contributes to the integration of AI with MBSE, focusing solely on the extraction and transformation of medical standards information from documents into SysML norm models. It explores the initial outcomes of augmenting data extraction from medical standards using recent AI algorithms and integrating them into MBSE practices. The evaluation involves two approaches, an open‐source multimodal classifier model and a proprietary large language model. The study assesses these approaches on a medical standard and outlines future work, including the exploration of an open‐source large language model approach.

IMG-20. IDENTIFICATION OF PRE-OPERATIVE IMAGING FEATURES ASSOCIATED WITH COGNITIVE IMPAIRMENT IN CHILDREN WITH BRAIN TUMOURS USING A LANGUAGE MODELLING SYSTEM COMBINED WITH MACHINE LEARNING ANALYSIS

Abstract BACKGROUND Cognitive impairments caused by paediatric brain tumours negatively impact future quality-of-life. Impairments of processing speed and working memory are important contributors to challenges these children encounter in education. Inconsistent image acquisition and poor imaging quality can restrict advanced radiomic analysis in historical datasets. We developed and refined a language modelling human image-analysis support system, passing data to specific machine learning techniques that can utilise datasets with many data points in a limited population. METHODS Medical records of children treated for brain tumours with surgery and radiotherapy in Manchester between 2000-2020 were reviewed with imaging accessible from 2006-2020. Pre-operative MRI scans were reviewed for 52 patients along with their most recent processing speed and/or working memory test scores. 274 radiological variables analysing tumour features and damage to brain structures were reported per patient using a standardised reporting tool. Univariate analysis was performed using Linear Regression with time-to-test analysed as a covariate. Multivariate analysis used Elastic Net Regression. RESULTS Univariate analysis revealed risk factors for slower processing speed included midline shift, tumour in the corpus callosum, right temporal periventricular white matter oedema, diffusion changes in the left frontal lobe or amygdala and FLAIR changes in the left caudate nucleus (p<0.005). Risk factors for impaired working memory included midline shift, cortical tumour location, left periventricular white matter oedema and diffusion or FLAIR changes in the left frontal lobe (p<0.005). Despite the limited population size and review of only pre-operative features, multivariate analysis produced multiple predictive models able to outperform chance when predicting processing speed and working memory outcomes. CONCLUSIONS This study provides evidence that a radiological language modelling approach can be utilised for outcome prediction. It identified imaging features predictive of cognitive impairments prior to surgery and radiotherapy. Further work is needed to assess inter-observer variability and validate these findings.

Automated Scoring of Scientific Creativity in German

ABSTRACTAutomated scoring is a current hot topic in creativity research. However, most research has focused on the English language and popular verbal creative thinking tasks, such as the alternate uses task. Therefore, in this study, we present a large language model approach for automated scoring of a scientific creative thinking task that assesses divergent ideation in experimental tasks in the German language. Participants are required to generate alternative explanations for an empirical observation. This work analyzed a total of 13,423 unique responses. To predict human ratings of originality, we used XLM‐RoBERTa (Cross‐lingual Language Model‐RoBERTa), a large, multilingual model. The prediction model was trained on 9,400 responses. Results showed a strong correlation between model predictions and human ratings in a held‐out test set (n = 2,682; r = 0.80; CI‐95% [0.79, 0.81]). These promising findings underscore the potential of large language models for automated scoring of scientific creative thinking in the German language. We encourage researchers to further investigate automated scoring of other domain‐specific creative thinking tasks.

Guiding a language-model based protein design method towards MHC Class-I immune-visibility targets in vaccines and therapeutics

Proteins have an arsenal of medical applications that include disrupting protein interactions, acting as potent vaccines, and replacing genetically deficient proteins. While therapeutics must avoid triggering unwanted immune-responses, vaccines should support a robust immune-reaction targeting a broad range of pathogen variants. Therefore, computational methods modifying proteins’ immunogenicity without disrupting function are needed. While many components of the immune-system can be involved in a reaction, we focus on Cytotoxic T-lymphocytes (CTLs). These target short peptides presented via the MHC Class I (MHC-I) pathway. To explore the limits of modifying the visibility of those peptides to CTLs within the distribution of naturally occurring sequences, we developed a novel machine learning technique, CAPE-XVAE. It combines a language model with reinforcement learning to modify a protein’s immune-visibility. Our results show that CAPE-XVAE effectively modifies the visibility of the HIV Nef protein to CTLs. We contrast CAPE-XVAE to CAPE-Packer, a physics-based method we also developed. Compared to CAPE-Packer, the machine learning approach suggests sequences that draw upon local sequence similarities in the training set. This is beneficial for vaccine development, where the sequence should be representative of the real viral population. Additionally, the language model approach holds promise for preserving both known and unknown functional constraints, which is essential for the immune-modulation of therapeutic proteins. In contrast, CAPE-Packer, emphasizes preserving the protein’s overall fold and can reach greater extremes of immune-visibility, but falls short of capturing the sequence diversity of viral variants available to learn from. Source code: https://github.com/hcgasser/CAPE (Tag: v1.1)

MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Classification Tasks

This paper presents a benchmark study of MIDI-domain music classification using the mask language modeling approach of the Bidirectional Encoder Representations from Transformers (BERT). Specifically, with five public-domain datasets of single-track polyphonic piano MIDI files, we pre-train a 12-layer Transformer model using the BERT approach and fine-tune it for four downstream classification tasks. These include two note-level classification tasks---melody extraction and velocity prediction, and two sequence-level classification tasks---artist classification and emotion classification. Compared to strong recurrent neural network (RNN)-based baselines, the BERT approach does lead to higher classification accuracy, with less than 10 epochs of fine-tuning. Ablation studies further show that the pre-training remains effective even if the MIDI data of the downstream tasks are not seen during the pre-training stage, and that freezing the self-attention layers at the fine-tuning stage only slightly degrades the performance. For reproducibility, we share the weights of our pre-trained and fine-tuned models.

An improved transformer‐based model for detecting phishing, spam and ham emails: A large language model approach

AbstractPhishing and spam have been a cybersecurity threat with the majority of breaches resulting from these types of social engineering attacks. Therefore, detection has been a long‐standing challenge for both academic and industry researcher. New and innovative approaches are required to keep up with the growing sophistication of threat actors. One such illumination which has vast potential are large language models (LLM). LLM emerged and already demonstrated their potential to transform society and provide new and innovative approaches to solve well‐established challenges. Phishing and spam have caused financial hardships and lost time and resources to email users all over the world and frequently serve as an entry point for ransomware threat actors. While detection approaches exist, especially heuristic‐based approaches, LLMs offer the potential to venture into a new unexplored area for understanding and solving this challenge. LLMs have rapidly altered the landscape from business, consumers, and throughout academia and demonstrate transformational potential to profoundly impact the society. Based on this, applying these new and innovative approaches to email detection is a rational next step in academic research. In this work, we present IPSDM, an improved phishing spam detection model based on fine‐tuning the BERT family of models to specifically detect phishing and spam emails. We demonstrate our fine‐tuned version, IPSDM, is able to better classify emails in both unbalanced and balanced datasets. Moreover, IPSDM consistently outperforms the baseline models in terms of classification accuracy, precision, recall, and F1‐score, while concurrently mitigating overfitting concerns.

Sensor event sequence prediction for proactive smart home: A GPT2-based autoregressive language model approach

We propose a framework for predicting sensor event sequences (SES) in smart homes, which can proactively support residents’ activities and alert them if activities are not completed as intended. We leverage ongoing activity recognition to enhance the prediction performance, employing a GPT2-based model typically used for sentence generation. We hypothesize that the relationship between ongoing activities and SES patterns is akin to the relationship between topics and word sequence patterns in natural language processing (NLP), enabling us to apply the GPT2-based model to SES prediction. We empirically evaluated our method using two real-world datasets in which residents performed their usual daily activities. Our experimental results demonstrates that the use of the GPT2-based model significantly improves the F1 value of SES prediction from 0.461 to 0.708 compared to the state-of-the-art method, and that leveraging knowledge on ongoing activity can further improve performance to 0.837. Achieving these SES predictions using the ongoing activity recognition model required simple feature engineering and modeling, yielding a performance rate of approximately 80%.

The Language Model Can Have the Personality: Joint Learning for Personality Enhanced Language Model (Student Abstract)

With the introduction of large language models, chatbots are becoming more conversational to communicate effectively and capable of handling increasingly complex tasks. To make a chatbot more relatable and engaging, we propose a new language model idea that maps the human-like personality. In this paper, we propose a systematic Personality-Enhanced Language Model (PELM) approach by using a joint learning mechanism of personality classification and language generation tasks. The proposed PELM leverages a dataset of defined personality typology, Myers-Briggs Type Indicator, and produces a Personality-Enhanced Language Model by using a joint learning and cross-teaching structure consisting of a classification and language modelling to incorporate personalities via both distinctive types and textual information. The results show that PELM can generate better personality-based outputs than baseline models.

Parameter-efficient fine-tuning large language model approach for hospital discharge paper summarization

Text summarization in medical domain is one of the most crucial chores as it deals with the critical human information. Consequently the proper summarization and key point extraction from medical deeds using pre-trained Language models is now the key figure to be focused on for the researchers. But due to the considerable amount of real-world data and enormous amount of memory requirement to train the Large Language Models (LLMs), research on these models become challenging. To overcome these challenges multiple prompting and tuning techniques are being used. In this paper, effectiveness of prompt engineering and parameter efficient fine tuning is being studied to summarize the Hospital Discharge Summary (HDS) papers effectively, so that these models can accurately interprete medical terminologies and contexts, generate brief but compact summaries, and draw out concentrated themes, which opens new approaches for the application of LLMs in healthcare and making HDS more patient-friendly. In this research LLaMA 2 (Large Language Model Meta AI) has been considered as the base model. Also, the model has been fine-tuned using QLoRA (Quantized Low Rank Adapters), which can bring down the memory usage of LLMs without compromising the data quality. This study explores the way to use LLMs on HDS datasets without the hassle of memory usage using QLoRA, into electronic health record systems to further streamline the handling and retrieval of healthcare information.

Abstract 4966: Machine learning and large language model approach to pancancer data elements

Abstract Introductory Statement: The goal is to use machine learning (ML) and large language model (LLM) to augment the manual curation of cancer data elements. Introduction: Memorial Sloan Kettering Cancer Center (MSKCC) has ~100,000 cancer patients and counting with genomic testing. Clinicians use genomic data for research but lack clinical data to analyze together. We use a vendor, VASTA Global to hire curators to manually curate cancer patient’s core clinical data elements (CCDE) within unstructured/paragraph text in electronic medical record (EMR) notes. CCDE encompasses 122 data elements that include a patient’s full cancer history that can take up to 1 working day to curate. We collaborated with the Realyze Intelligence Healthcare Solutions vendor to use their AI pipeline to generate the manual curated dataset. Realyze generated the CCDE data elements such as histology, pathology site, MMR, TNM staging, ECOG, and KPS for a pilot lung cancer cohort of 150 patients. We manually validated the generated data for 74 out of 150 patients. Methods:The Realyze platform uses a combination of LLMs, ML algorithms and standard terminologies to create a cancer patient model. These models are flexible enough to address the unique needs and challenges of a pan-cancer oncology model. By using standardized FHIR export, results were delivered to a data lake solution and written into a REDCap database to enable human review. Summary:We manually assessed 74 patients. The NLP gave concordant values for MMR, KPS and TNM staging for 100% of the instances. For MMR these were all null values with false negative (FN) of 100% accuracy. Pathology site had 92.15% accuracy while histology has 97.5% accuracy. Conclusion:Will work on refining pathology site and histology’s ICDO3 list to increase the percentage of accuracy. Once Realyze refines their model for these data elements we will re-run it on a larger cohort of cancer patients and calculate the accuracy. Accuracy Results Clinical data elements 74 patients assessed: Accuracy % ECOG 98.6 KPS 100 T (path) 100 T (clinical) 100 N (path) 100 N (clinical) 100 M (path) 100 M(clinical) 100 MMR 100 Histology (path) 97.5 Path site 92.15 Citation Format: Andrew Niederhausern, Nadia S. Bahadur, Gary Wallace, Gilan E. Saadawi, John Philip. Machine learning and large language model approach to pancancer data elements [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4966.

NanoBERTa-ASP: predicting nanobody paratope based on a pretrained RoBERTa model

BackgroundNanobodies, also known as VHH or single-domain antibodies, are unique antibody fragments derived solely from heavy chains. They offer advantages of small molecules and conventional antibodies, making them promising therapeutics. The paratope is the specific region on an antibody that binds to an antigen. Paratope prediction involves the identification and characterization of the antigen-binding site on an antibody. This process is crucial for understanding the specificity and affinity of antibody-antigen interactions. Various computational methods and experimental approaches have been developed to predict and analyze paratopes, contributing to advancements in antibody engineering, drug development, and immunotherapy. However, existing predictive models trained on traditional antibodies may not be suitable for nanobodies. Additionally, the limited availability of nanobody datasets poses challenges in constructing accurate models.MethodsTo address these challenges, we have developed a novel nanobody prediction model, named NanoBERTa-ASP (Antibody Specificity Prediction), which is specifically designed for predicting nanobody-antigen binding sites. The model adopts a training strategy more suitable for nanobodies, based on an advanced natural language processing (NLP) model called BERT (Bidirectional Encoder Representations from Transformers). To be more specific, the model utilizes a masked language modeling approach named RoBERTa (Robustly Optimized BERT Pretraining Approach) to learn the contextual information of the nanobody sequence and predict its binding site.ResultsNanoBERTa-ASP achieved exceptional performance in predicting nanobody binding sites, outperforming existing methods, indicating its proficiency in capturing sequence information specific to nanobodies and accurately identifying their binding sites. Furthermore, NanoBERTa-ASP provides insights into the interaction mechanisms between nanobodies and antigens, contributing to a better understanding of nanobodies and facilitating the design and development of nanobodies with therapeutic potential.ConclusionNanoBERTa-ASP represents a significant advancement in nanobody paratope prediction. Its superior performance highlights the potential of deep learning approaches in nanobody research. By leveraging the increasing volume of nanobody data, NanoBERTa-ASP can further refine its predictions, enhance its performance, and contribute to the development of novel nanobody-based therapeutics.Github repository: https://github.com/WangLabforComputationalBiology/NanoBERTa-ASP

Arabic sarcasm detection: An enhanced fine-tuned language model approach

Sarcasm is a complex linguistic phenomenon involving humor, criticism, or phrases that convey the opposite meaning, mask true feelings, and play pivotal roles in various aspects of communication. Therefore, identifying sarcasm is essential for sentiment analysis, social media monitoring, and customer service, as it enables a better understanding of public sentiment. Moreover, social media has become a primary platform for people to express their feelings and opinions and provide feedback to businesses and service providers. Misinterpreting sarcasm in customer feedback can lead to incorrect responses and actions. However, accurately detecting sarcasm is challenging because it depends on context, cultural factors, and inherent ambiguity. Despite the plenty of research and resources in Machine Learning (ML) for detecting sarcasm in English, including Deep Learning (DL) techniques, there is still a shortage of research in sarcasm detection in Arabic, particularly in DL methodologies and available sarcastic datasets. This paper constructed a new Arabic sarcastic corpus and fine-tuned three pre-trained Arabic transformer-based Language Models (LM) for Arabic sarcasm detection. We also proposed a hybrid DL approach for sarcasm detection that combines static and contextualized representations using pre-trained LM, such as Word2Vec word embeddings and Bidirectional Encoder Representations from Transformers (BERT) models pretrained on Arabic resources. The proposed enhanced hybrid deep learning approach outperforms state-of-the-art models by 8% on a shared benchmark dataset and achieves a 5% improvement in F1-score on another.