Natural Language Processing Learning Research Articles

Multi-step Transfer Learning in Natural Language Processing for the Health Domain

The restricted access to data in healthcare facilities due to patient privacy and confidentiality policies has led to the application of general natural language processing (NLP) techniques advancing relatively slowly in the health domain. Additionally, because clinical data is unique to various institutions and laboratories, there are not enough standards and conventions for data annotation. In places without robust death registration systems, the cause of death (COD) is determined through a verbal autopsy (VA) report. A non-clinician field agent completes a VA report using a set of standardized questions as guide to identify the symptoms of a COD. The narrative text of the VA report is used as a case study to examine the difficulties of applying NLP techniques to the healthcare domain. This paper presents a framework that leverages knowledge across multiple domains via two domain adaptation techniques: feature extraction and fine-tuning. These techniques aim to improve VA text representations for COD classification tasks in the health domain. The framework is motivated by multi-step learning, where a final learning task is realized via a sequence of intermediate learning tasks. The framework builds upon the strengths of the Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Models (ELMo) models pretrained on the general English and biomedical domains. These models are employed to extract features from the VA narratives. Our results demonstrate improved performance when initializing the learning of BERT embeddings with ELMo embeddings. The benefit of incorporating character-level information for learning word embeddings in the English domain, coupled with word-level information for learning word embeddings in the biomedical domain, is also evident.

Fast Recurrent Neural Network with Bi-LSTM for Handwritten Tamil Text Segmentation in NLP

Tamil text segmentation is a long-standing test in language comprehension that entails separating a record into adjacent pieces based on its semantic design. Each segment is important in its own way. The segments are organised according to the purpose of the content examination as text groups, sentences, phrases, words, characters or any other data unit. That process has been portioned using rapid tangled neural organisation in this research, which presents content segmentation methods based on deep learning in natural language processing (NLP). This study proposes a bidirectional long short-term memory (Bi-LSTM) neural network prototype in which fast recurrent neural networks (FRNNs) are used to learn Tamil text group embedding and phrases are fragmented using text-oriented data. As a result, this prototype is capable of handling variable measured setting data and gives a vast new dataset for naturally segmenting text in Tamil. In addition, we develop a segmentation prototype and show how well it sums up to unnoticeable regular content using this dataset as a base. With Bi-LSTM, the segmentation precision of FRNN is superior to that of other segmentation approaches; however, it is still inferior to that of certain other techniques. Every content is scaled to the required size in the proposed framework, which is immediately accessible for the preparation. This means, each word in a scaled Tamil text is employed to prepare neural organisation as fragmented content. The results reveal that the proposed framework produces high rates of segmentation for manually authored material that are nearly equivalent to segmentation-based plans.

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

BackgroundPretraining large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing. With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from free text has improved significantly in both the general and medical domains. However, it is difficult to train specific BERT models to perform well in domains for which few databases of a high quality and large size are publicly available. ObjectiveWe hypothesized that this problem could be addressed by oversampling a domain-specific corpus and using it for pretraining with a larger corpus in a balanced manner. In the present study, we verified our hypothesis by developing pretraining models using our method and evaluating their performance. MethodsOur proposed method was based on the simultaneous pretraining of models with knowledge from distinct domains after oversampling. We conducted three experiments in which we generated (1) English biomedical BERT from a small biomedical corpus, (2) Japanese medical BERT from a small medical corpus, and (3) enhanced biomedical BERT pretrained with complete PubMed abstracts in a balanced manner. We then compared their performance with those of conventional models. ResultsOur English BERT pretrained using both general and small medical domain corpora performed sufficiently well for practical use on the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our proposed method was more effective than the conventional methods for each biomedical corpus of the same corpus size in the general domain. Our Japanese medical BERT outperformed the other BERT models built using a conventional method for almost all the medical tasks. The model demonstrated the same trend as that of the first experiment in English. Further, our enhanced biomedical BERT model, which was not pretrained on clinical notes, achieved superior clinical and biomedical scores on the BLUE benchmark with an increase of 0.3 points in the clinical score and 0.5 points in the biomedical score. These scores were above those of the models trained without our proposed method. ConclusionsWell-balanced pretraining using oversampling instances derived from a corpus appropriate for the target task allowed us to construct a high-performance BERT model.

Assessment of Gender Differences in Letters of Recommendation for Physical Therapy Residency Applications.

Letters of recommendation (LOR) are an integral component of physical therapy residency applications. Identifying the influence of applicant and writer gender in LOR will help identify whether potential implicit gender bias exists in physical therapy residency application processes. Several medical and surgical residency education programs have reported positive, neutral, or negative LOR female gender bias among applicants and writers. Little research exists on gender differences in LOR to physical therapy education programs or physical therapy residency programs. Seven hundred sixty-eight LOR were analyzed from 256 applications to 3 physical therapy residency programs (neurologic, orthopaedic, sports) at one institution from 2014 to 2020. Thematic categories were developed to identify themes in a sample of LOR. Associations between writer and applicant gender were analyzed using summary statistics, word counts, thematic and psycholinguistic extraction, and rule-based and deep learning Natural Language Processing . No significant difference in LOR word counts were found based on writer or applicant gender. Increased word counts were seen in sports residency LOR compared with the orthopaedic residency. Thematic analysis showed LOR gender differences with male applicants receiving more positive generalized recommendations and female applicants receiving more comments regarding interpersonal relationship skills. No thematic or psycholinguistic gender differences were seen by LOR writer. Male applicants were 1.9 times more likely to select all male LOR writers, whereas female applicants were 2.1 times more likely to choose all female LOR writers. Gender differences in LORs for physical therapy residencies were found using a comprehensive Natural Language Processing approach that identified both a positive recommendation male applicant gender bias and a positive interpersonal relationship skill female applicant gender bias. Applicants were not harmed nor helped by selecting LOR writers of the opposite gender. Admissions committees and LOR writers should be mindful of potential implicit gender biases in LOR submitted to physical therapy residency programs.

GTLNLP: A Mathematical Exploration of Cross-Domain Knowledge Transfer for Text Generation for Generative Transfer Learning in Natural Language Processing

In the field of Cross-Domain Knowledge Transfer generative transfer learning in Natural Language Processing (NLP) used for creating text using generative math. The paper aims to improve the usefulness of text creation models in a variety of areas by using cutting edge deep learning and neural network methods. We come up with a new system that makes it easier for information to move from one domain to another, even when the language and situation are different in each domain. Using domain adaptation techniques to match feature distributions and reduce domain gaps is what our method is based on mathematically. It is a complex version of transfer learning principles. We test our model's abilities on a wide range of tasks by doing a lot of careful experiments. We focus on how well it can share information and write text that makes sense and is relevant to the situation across different areas. This study not only adds to our theoretical understanding of cross-domain knowledge transfer, but it also gives us useful tips on how to make NLP models more flexible and useful in the real world. The results of our study could help improve the state of the art in generative transfer learning and make text generation systems that work better and more reliably in a variety of language settings.

Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis

AbstractVenous thromboembolism (VTE) is a leading cause of preventable in-hospital mortality. Monitoring VTE cases is limited by the challenges of manual medical record review and diagnosis code interpretation. Natural language processing (NLP) can automate the process. Rule-based NLP methods are effective but time consuming. Machine learning (ML)-NLP methods present a promising solution. We conducted a systematic review and meta-analysis of studies published before May 2023 that use ML-NLP to identify VTE diagnoses in the electronic health records. Four reviewers screened all manuscripts, excluding studies that only used a rule-based method. A meta-analysis evaluated the pooled performance of each study’s best performing model that evaluated for pulmonary embolism and/or deep vein thrombosis. Pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with confidence interval (CI) were calculated by DerSimonian and Laird method using a random-effects model. Study quality was assessed using an adapted TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) tool. Thirteen studies were included in the systematic review and 8 had data available for meta-analysis. Pooled sensitivity was 0.931 (95% CI, 0.881-0.962), specificity 0.984 (95% CI, 0.967-0.992), PPV 0.910 (95% CI, 0.865-0.941) and NPV 0.985 (95% CI, 0.977-0.990). All studies met at least 13 of the 21 NLP-modified TRIPOD items, demonstrating fair quality. The highest performing models used vectorization rather than bag-of-words and deep-learning techniques such as convolutional neural networks. There was significant heterogeneity in the studies, and only 4 validated their model on an external data set. Further standardization of ML studies can help progress this novel technology toward real-world implementation.

Improving the performance of machine learning penicillin adverse drug reaction classification with synthetic data and transfer learning.

Machine learning may assist with the identification of potentially inappropriate penicillin allergy labels. Strategies to improve the performance of existing models for this task include the use of additional training data, synthetic data and transfer learning. The aims of this study were to investigate the use of additional training data and novel machine learning strategies, namely synthetic data and transfer learning, to improve the performance of penicillin adverse drug reaction (ADR) machine learning classification. Machine learning natural language processing was applied to free-text penicillin ADR data extracted from a public health system electronic health record (EHR). The models were developed by training on various labelled data sets. ADR entries were split into training and testing data sets and used to develop and test a variety of machine learning models. The effect of training on additional data and synthetic data versus the use of transfer learning was analysed. Following the application of these techniques, the area under the receiver operator curve of best-performing models for the classification of penicillin allergy (vs intolerance) and high-risk allergy (vs low-risk allergy) improved to 0.984 (using the artificial neural network model) and 0.995 (with the transfer learning approach) respectively. Machine learning models demonstrate high levels of accuracy in the classification and risk stratification of penicillin ADR labels using the reaction documented in the EHR. The model can be further optimised by incorporating additional training data and using transfer learning. Practical applications include automating case detection for penicillin allergy delabelling programmes.

Application of self-supervised learning in natural language processing

Self-supervised learning uses the label-free data learning model and has a significant impact on the NLP task. It reduces data annotation costs and improves performance. The main applications include pre-training models such as BERT and GPT, contrast learning, and pseudo-supervised and semi-supervised methods. It has been successfully applied in text classification, emotion analysis and other fields. Future research directions include mixed unsupervised learning, cross-modal learning and improving interpretability of models while focusing on ethical social issues.

A hybrid combination of CNN Attention with optimized random forest with grey wolf optimizer to discriminate between Arabic hateful, abusive tweets

Arabic hateful speech recognition has long been a major area of focus in Natural Language Processing (NLP) research. In light of recent advancements in transformer models and deep learning, researchers are now turning to transfer learning techniques based on existing models such as BERT for Arabic hateful speech recognition. To detect Arabic hateful contexts, using advanced machine learning algorithms and NLP techniques is essential. These techniques can help to detect different forms of hateful contexts in Arabic by analyzing the text for lexical, semantic, and syntactic features. In this research, we proposed a new hybrid approach that combines deep and machine learning models to detect hateful and abusive content in Arabic. The proposed model consists of a combination of convolutional neural networks and attention layers that are trained to differentiate between normal, abusive, and hateful contexts in Arabic. In the first step, we used a pre-trained model to extract features from the hateful Arabic context. After that, we used an optimized random forest combined with particle swarm optimization and grey wolf optimizer to classify the extracted features. Finally, we evaluated the performance of the model to detect hateful Arabic contexts. To evaluate the proposed method we used 5846 and 6023 tweets with 3 categories of hateful, abusive, and normal Arabic contexts. The experimental result indicates 97.16% accuracy, 97.15% F1-score, 97.17% precision, and 97.13% sensitivity using CNN Attention + optimized random forest by the grey wolf optimizer on 5846 tweets. 97.83% accuracy, 97.83% F1-score, 97.84% precision, and 97.83% sensitivity have been reported CNN Attention + optimized random forest by the grey wolf optimizer on 6023 tweets.

CONSMI: Contrastive Learning in the Simplified Molecular Input Line Entry System Helps Generate Better Molecules.

In recent years, the application of deep learning in molecular de novo design has gained significant attention. One successful approach involves using SMILES representations of molecules and treating the generation task as a text generation problem, yielding promising results. However, the generation of more effective and novel molecules remains a key research area. Due to the fact that a molecule can have multiple SMILES representations, it is not sufficient to consider only one of them for molecular generation. To make up for this deficiency, and also motivated by the advancements in contrastive learning in natural language processing, we propose a contrastive learning framework called CONSMI to learn more comprehensive SMILES representations. This framework leverages different SMILES representations of the same molecule as positive examples and other SMILES representations as negative examples for contrastive learning. The experimental results of generation tasks demonstrate that CONSMI significantly enhances the novelty of generated molecules while maintaining a high validity. Moreover, the generated molecules have similar chemical properties compared to the original dataset. Additionally, we find that CONSMI can achieve favorable results in classifier tasks, such as the compound-protein interaction task.

AI-Assisted Summarization of Radiologic Reports: Evaluating GPT3davinci, BARTcnn, LongT5booksum, LEDbooksum, LEDlegal, and LEDclinical.

The review of clinical reports is an essential part of monitoring disease progression. Synthesizing multiple imaging reports is also important for clinical decisions. It is critical to aggregate information quickly and accurately. Machine learning natural language processing (NLP) models hold promise to address an unmet need for report summarization. We evaluated NLP methods to summarize longitudinal aneurysm reports. A total of 137 clinical reports and 100 PubMed case reports were used in this study. Models were 1) compared against expert-generated summary using longitudinal imaging notes collected in our institute and 2) compared using publicly accessible PubMed case reports. Five AI models were used to summarize the clinical reports, and a sixth model, the online GPT3davinci NLP large language model (LLM), was added for the summarization of PubMed case reports. We assessed the summary quality through comparison with expert summaries using quantitative metrics and quality reviews by experts. In clinical summarization, BARTcnn had the best performance (BERTscore = 0.8371), followed by LongT5Booksum and LEDlegal. In the analysis using PubMed case reports, GPT3davinci demonstrated the best performance, followed by models BARTcnn and then LEDbooksum (BERTscore = 0.894, 0.872, and 0.867, respectively). AI NLP summarization models demonstrated great potential in summarizing longitudinal aneurysm reports, though none yet reached the level of quality for clinical usage. We found the online GPT LLM outperformed the others; however, the BARTcnn model is potentially more useful because it can be implemented on-site. Future work to improve summarization, address other types of neuroimaging reports, and develop structured reports may allow NLP models to ease clinical workflow.

Assessing the utility of deep neural networks in detecting superficial surgical site infections from free text electronic health record data.

High-quality outcomes data is crucial for continued surgical quality improvement. Outcomes are generally captured through structured administrative data or through manual curation of unstructured electronic health record (EHR) data. The aim of this study was to apply natural language processing (NLP) to chart notes in the EHR to accurately capture postoperative superficial surgical site infections (SSSIs). Deep Learning (DL) NLP models were trained on data from 389,865 surgical cases across all 11 hospitals in the Capital Region of Denmark. Surgical cases in the training dataset were performed between January 01st, 2017, and October 30th, 2021. We trained a forward reading and a backward reading universal language model on unlabeled postoperative chart notes recorded within 30 days of a surgical procedure. The two language models were subsequently finetuned on labeled data for the classification of SSSIs. Validation and testing were performed on surgical cases performed during the month of November 2021. We propose two different use cases: a stand-alone machine learning (SAM) pipeline and a human-in-the-loop (HITL) pipeline. Performances of both pipelines were compared to administrative data and to manual curation. The models were trained on 3,983,864 unlabeled chart notes and finetuned on 1,231,656 labeled notes. Models had a test area under the receiver operating characteristic curves (ROC AUC) of 0.989 on individual chart notes and 0.980 on an aggregated case level. The SAM pipeline had a sensitivity of 0.604, a specificity of 0.996, a positive predictive value (PPV) of 0.763, and a negative predictive value (NPV) of 0.991. Prior to human review, the HITL pipeline had a sensitivity of 0.854, a specificity of 0.987, a PPV of 0.603, and a NPV of 0.997. The performance of the SAM pipeline was superior to administrative data, and significantly outperformed previously published results. The performance of the HITL pipeline approached that of manual curation.

A Study on the Application of Seq2seq in English Flipped Classroom Teaching in Colleges and Universities

Abstract With the rapid development of educational technology, the English teaching mode in colleges and universities faces innovation challenges. As an emerging teaching mode, flipped classroom promotes students’ active learning and improves classroom efficiency by switching the order of pre-course learning and in-class activities.Seq2seq technology, as a significant application of deep learning in natural language processing, is used in flipped classroom to provide new perspectives and methods for English teaching, which is expected to stimulate students’ interest in learning and improve the quality of teaching through technological means. This study aims to examine Seq2Seq2’s ability to enhance the quality of teaching and learning. This study explores the impact of Seq2seq technology on English flipped classroom teaching in colleges and universities. Using an experimental research method, 120 first-year university students were selected as the research subjects and divided into experimental and control groups, applying Seq2seq-assisted flipped classroom teaching and traditional teaching methods respectively. The study used a pre-test and post-test design, and collected students’ English achievement and learning motivation questionnaires to analyze data. Students in the experimental group outperformed the control group in English, with a mean score increase of 6.27 points, and their motivation to learn was also improved. The students in the experimental group showed improvement in vocabulary mastery, reading comprehension, and writing ability. The study shows that applying Seq2seq technology in flipped classroom can effectively improve students’ English learning achievement and learning motivation, which is of great significance for promoting the development of English teaching in colleges and universities.

Taxonomy for Deep Learning in Natural Language Processing

Taxonomy for Deep Learning in Natural Language Processing

Improving Sentiment Classification on Restaurant Reviews Using Deep Learning Models

Natural Language processing (NLP) includes the task of classifying texts. The proposed paper uses the ability and power of machine learning and deep learning techniques for sentiment analysis on restaurant reviews. With the increasing use of social media, classification of reviews has significant role in the development of public opinion. This work has extensive applications from client opinion study, market research and social media monitoring. The machine learning algorithms and NLP techniques are used classifying reviews and the results are rigorously analysed. It was noted that results obtained from logistic regression and multinomial naïve bayes algorithms are better than other machine learning models. Hence these two algorithms are selected to optimize further. After tuning the hyperparameters by grid search technique, the best accuracy obtained by logistic regression and multinomial naive bayes on validation dataset was 89.9% and 89.6% respectively. Deep learning architectures were also utilized for reviews classification. The accuracy obtained by Convolutional Neural Net (CNN) and Bi-Directional Long-short-term-memory (Bi-LSTM) was 89% and 90% respectively.

An Integrated Voice Recognition and Natural Language Processing Platform to Automatically Extract Thoracolumbar Injury Classification Score Features From Radiology Reports

Many predictive models for estimating clinical outcomes after spine surgery have been reported in the literature. However, implementation of predictive scores in practice is limited by the time-intensive nature of manually abstracting relevant predictors. In this study, we designed natural language processing (NLP) algorithms to automate data abstraction for the thoracolumbar injury classification score (TLICS). We retrieved the radiology reports of all Mayo Clinic patients with an International Classification of Diseases, 9th or 10th revision, code corresponding to a fracture of the thoracolumbar spine between January 2005 and October 2020. Annotated data were used to train an N-gram NLP model using machine learning methods, including random forest, stepwise linear discriminant analysis, k-nearest neighbors, and penalized logistic regression models. A total of 1085 spine radiology reports were included in our analysis. Our dataset included 483 compression, 401 burst, 103 translational/rotational, and 98 distraction fractures. A total of 103 reports had documented an injury of the posterior ligamentous complex. The overall accuracy of the random forest model for fracture morphology feature detection was 76.96% versus 65.90% in the stepwise linear discriminant analysis, 50.69% in the k-nearest neighbors, and 62.67% in the penalized logistic regression. The overall accuracy to detect posterior ligamentous complex integrity was highest in the random forest model at 83.41%. Our random forest model was implemented in the backend of a web application in which users can dictate reports and have TLICS features automatically extracted. We have developed a machine learning NLP model for extracting TLICS features from radiology reports, which we deployed in a web application that can be integrated into clinical practice.

FVP: Fourier Visual Prompting for Source-Free Unsupervised Domain Adaptation of Medical Image Segmentation.

Medical image segmentation methods normally perform poorly when there is a domain shift between training and testing data. Unsupervised Domain Adaptation (UDA) addresses the domain shift problem by training the model using both labeled data from the source domain and unlabeled data from the target domain. Source-Free UDA (SFUDA) was recently proposed for UDA without requiring the source data during the adaptation, due to data privacy or data transmission issues, which normally adapts the pre-trained deep model in the testing stage. However, in real clinical scenarios of medical image segmentation, the trained model is normally frozen in the testing stage. In this paper, we propose Fourier Visual Prompting (FVP) for SFUDA of medical image segmentation. Inspired by prompting learning in natural language processing, FVP steers the frozen pre-trained model to perform well in the target domain by adding a visual prompt to the input target data. In FVP, the visual prompt is parameterized using only a small amount of low-frequency learnable parameters in the input frequency space, and is learned by minimizing the segmentation loss between the predicted segmentation of the prompted target image and reliable pseudo segmentation label of the target image under the frozen model. To our knowledge, FVP is the first work to apply visual prompts to SFUDA for medical image segmentation. The proposed FVP is validated using three public datasets, and experiments demonstrate that FVP yields better segmentation results, compared with various existing methods.

ProteinMAE: masked autoencoder for protein surface self-supervised learning.

The biological functions of proteins are determined by the chemical and geometric properties of their surfaces. Recently, with the booming progress of deep learning, a series of learning-based surface descriptors have been proposed and achieved inspirational performance in many tasks such as protein design, protein-protein interaction prediction, etc. However, they are still limited by the problem of label scarcity, since the labels are typically obtained through wet experiments. Inspired by the great success of self-supervised learning in natural language processing and computer vision, we introduce ProteinMAE, a self-supervised framework specifically designed for protein surface representation to mitigate label scarcity. Specifically, we propose an efficient network and utilize a large number of accessible unlabeled protein data to pretrain it by self-supervised learning. Then we use the pretrained weights as initialization and fine-tune the network on downstream tasks. To demonstrate the effectiveness of our method, we conduct experiments on three different downstream tasks including binding site identification in protein surface, ligand-binding protein pocket classification, and protein-protein interaction prediction. The extensive experiments show that our method not only successfully improves the network's performance on all downstream tasks, but also achieves competitive performance with state-of-the-art methods. Moreover, our proposed network also exhibits significant advantages in terms of computational cost, which only requires less than a tenth of memory cost of previous methods. https://github.com/phdymz/ProteinMAE.

A Transformer Natural Language Processing Algorithm for Cancer Associated Thrombosis Phenotype

Introduction: Cancer patients have high risk of venous thromboembolism (VTE). The ability to accurately capture VTE outcome in large electronic health record (EHR) is critical for both surveillance and risk stratification. We previously published a rule-based natural language processing (NLP) algorithm that relied on radiology reports alone with a precision (positive predictive value) of 89% and recall (sensitivity) of 69-78% (sTable 5 PMID36626707). The low recall metric was expected as VTE diagnosis that occur at outside hospital would not be captured by radiology reports. In the current study, we aimed to assess the performance of a machine learning NLP algorithm using both radiology and medical notes to detect cancer associated VTE longitudinally in a large cancer cohort. Methods: VTEwasdefined as symptomatic or incidental pulmonary embolism, lower or upper extremity deep vein thrombosis. Unusual thrombosis such as splanchnic vein thrombosis (mostly tumor thrombi) was classified as negative. Figure 1 details the study design: i) patient selection, ii) pre-annotation filtering, iii) gold standard annotation, iv) transformer model development. From a cohort of 9,769 patients with active cancer receiving first-line systemic therapy, we first pre-filtered all clinic progress notes, discharge summaries, and radiology reports to exclude sections with little clinical relevance (e.g. review of systems, physical exam), while maintaining sections with high clinical value (e.g. history of present illness, assessment/plan, hospital problems/course). Then, we kept notes with VTE keyword stems (e.g. VTE, PE, DVT, thromb, embol, clot, filling defect, phlebitis). We selected 808 patients with at least 1 keyword to determine the gold standard VTE date and location through independent chart review process. This cohort was randomly split into a training and testing cohort at an 7:1 ratio. In the training cohort (n=708), we further annotated all keyword-containing phrases within 30 days (d) from VTE date and from index date until 180d if no VTE (n=~2,000). The ~800 positive and ~1,200 negative keyword-containing phrases were used to train ClinicalBERT (bidirectional encoder representation transformer) large language model through split-sample cross validation to finetune the loss function and to determine the prediction threshold. Finally, we applied the entire trained model package to every available notes/reports (n=~2300) in the test cohort (n=100) to assess if the first positively predicted date would correlate with the gold standard VTE date. Results: The finetuned ClinicalBERT modelachieved 98% precision and 98% recall on the patient level in the training cohort (n=708). When applied on the 100-patient test cohort, the model accurately predicted first VTE events in 50/54 patients (93% precision) and captured 50/54 of patients with VTE (93% recall sensitivity) (Table 1). Most of the tumor thrombi and superficial venous thrombi were correctly classified as negative. The false positives were related to unusual sentences with both positive and negative findings such as “Thrombosis surrounding the left brachial catheter in basilic vein,” “CT chest with PE was done,” “Lovenox was initiated for UE DVT, but this was port thrombosis in the basilic vein.” The false negatives were related to isolated VTE diagnosis close to death or loss to follow-up. Notably, the precision and recall tradeoff can be further finetuned by changing the minimum number of positively predicted notes from the NLP. Conclusion: In conclusion, we successfully developed and tested a transformer NLP model to detect cancer associated VTE longitudinally using a combination of clinical notes and radiology reports that achieved a precision of 93% and recall of 93%. This represented an improvement over our original algorithm with radiology report alone. Further validation of the model in external cohorts and finetuning through federated or transferred learning are ongoing to ensure model generalizability and usability.

Fine-Tuning IndoBERT for Indonesian Exam Question Classification Based on Bloom's Taxonomy

Background: The learning assessment of elementary schools has recently incorporated Bloom's Taxonomy, a structure in education that categorizes different levels of cognitive learning and thinking skills, as a fundamental framework. This assessment now includes High Order Thinking Skill (HOTS) questions, with a specific focus on Indonesian topics. The implementation of this system has been observed to require teachers to manually categorize or classify questions, and this process typically requires more time and resources. To address the associated difficulty, automated categorization and classification are required to streamline the process. However, despite various research efforts in questions classification, there is still room for improvement in terms of performance, particularly in precision and accuracy. Numerous investigations have explored the use of Deep Learning Natural Language Processing models such as BERT for classification, and IndoBERT is one such pre-trained model for text analysis. Objective: This research aims to build classification system that is capable of classifying Indonesian exam questions in multiple-choice form based on Bloom's Taxonomy using IndoBERT pre-trained model. Methods: The methodology used includes hyperparameter fine-tuning, which was carried out to identify the optimal model performance. This performance was subsequently evaluated based on accuracy, F1 Score, Precision, Recall, and the time required for the training and validation of the model. Results: The proposed Fine Tuned IndoBERT Model showed that the accuracy rate was 97%, 97% F1 Score, 97% Recall, and 98% Precision with an average training time per epoch of 1.55 seconds and an average validation time per epoch of 0.38 seconds. Conclusion: Fine Tuned IndoBERT model was observed to have a relatively high classification performance, and based on this observation, the system was considered capable of classifying Indonesian exam questions at the elementary school level. Keywords: IndoBERT, Fine Tuning, Indonesian Exam Question, Model Classifier, Natural Language Processing, Bloom’s Taxonomy