Pre-trained Bidirectional Encoder Representations Research Articles

Language conditioned multi-scale visual attention networks for visual grounding

Visual grounding (VG) is a task that requires to locate a specific region in an image according to a natural language expression. Existing efforts on the VG task are divided into two-stage, one-stage and Transformer-based methods, which have achieved high performance. However, most of the previous methods extract visual information at a single spatial scale and ignore visual information at other spatial scales, which makes these models unable to fully utilize the visual information. Moreover, the insufficient utilization of linguistic information, especially failure to capture global linguistic information, may lead to failure to fully understand language expressions, thus limiting the performance of these models. To better address the task, we propose a language conditioned multi-scale visual attention network (LMSVA) for visual grounding, which can sufficiently utilize visual and linguistic information to perform multimodal reasoning, thus improving performance of model. Specifically, we design a visual feature extractor containing a multi-scale layer to get the required multi-scale visual features by expanding the original backbone. Moreover, we exploit pooling the output of the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to extract sentence-level linguistic features, which can enable the model to capture global linguistic information. Inspired by the Transformer architecture, we present the Visual Attention Layer guided by Language and Multi-Scale Visual Features (VALMS), which is able to better learn the visual context guided by multi-scale visual and linguistic features, and facilitates further multimodal reasoning. Extensive experiments on four large benchmark datasets, including ReferItGame, RefCOCO, RefCOCO+ and RefCOCOg, demonstrate that our proposed model achieves the state-of-the-art performance.

Comparative Analysis of Deep Learning Models for Part of Speech Tagging in the Malay Language

Despite the widespread use of Malay, under-resourced languages like Malay face challenges in Natural Language Processing (NLP), particularly in Part-of-Speech (POS) tagging. The scarcity of annotated corpora poses a primary obstacle to POS tagging in Malay. This study aims to enhance the effectiveness and reliability of POS tagging models explicitly tailored for under-resourced languages within the field of NLP, focusing on Malay. Existing models, which rely on Conditional Random Fields and Hidden Markov Models, exhibit limitations, underscoring the need for more robust approaches. The research conducts a comparative analysis of various deep-learning models with different encoders for POS tagging in Malay sentences. The experimental analysis demonstrates that the Bidirectional Long Short-Term Memory (Bi-LSTM) model, leveraging a pre-trained Bidirectional Encoder Representations from Transformers (BERT) embedding model, achieves exceptional accuracy, precision, recall, and F1 scores in predicting tags. Notably, the BERT + Bi-LSTM model, boasting an accuracy of 98.82%, outperforms other models, showcasing superior performance across all evaluated metrics. Additionally, this combined model effectively handles known and unknown words, yielding highly accurate POS tagging results for Malay sentences. Doi: 10.28991/HIJ-2024-05-02-04 Full Text: PDF

Tales of Turbulence: BERT-based Multimodal Analysis of FED Communication Dynamics Amidst COVID-19 Through FOMC Minutes

AbstractThis study analyzes Federal Open Market Committee (FOMC) minutes using state-of-the-art Natural Language Processing techniques. We sought to investigate the effect of the global COVID-19 crisis on the FOMC minutes’ pattern and the strength of the Federal Reserve to influence inflation expectations through its primary press releases. To this end, we first quantified minutes leveraging domain-specific pre-trained Bidirectional Encoder Representations from Transformers models (FinBERTs). Then, we applied dynamic time warping to measure temporal sequence proximity over time. In addition, we built multivariable autoregressive integrated moving average models to verify our findings by injecting exogenous variables as indicator functions into the time series. The results suggest that the Federal Reserve has abstained from adjusting its tone and the forward-lookingness settings of its statements for the global pandemic. Therefore, the longstanding association of the FED's tone and forward-lookingness with consumers’ inflation expectations index has weakened during the global health crisis.

Abstract 4965: Detection of status of cancer in radiology notes using artificial intelligence

Abstract Introductory Statement: The goal is to use an AI model to replicate the human curator’s response on cancer status in radiology notes of ~20,000 cancer patients. Introduction: MSKCC currently has ~100,000 patients with genomic testing (IMPACT) and continues to accrue more. Clinicians use this genomic data for research but lack clinical structured data to analyze alongside the genomic data. We use a vendor called VASTA Global to manually curate unstructured paragraph text. Depending on the data model, we found that a patient’s full cancer history can take up to 8 hours to curate. Therefore, we want to implement AI to automate the curation of this data to save time and cost. We hope to achieve a faster curation process that allows us to accomplish more than 1 patient a day to catch up to the 100,000 MSK-IMPACT cohort. To decrease the average curation time per patient, we investigated the use of an NLP model to replicate manual curation of the PRISSMM™ ontology field for change in cancer status from radiology reports. Manual curation of radiology reports is time intensive due to curators understanding their cancer status and the volume of scans patients undergo. Methods:A pre-trained Bidirectional Encoder Representations from Transformers (BERT) model is fine-tuned using training data using GPUs in IBM Cloud Pak for Data (CPD) platform. The hyperparameters were adjusted using accuracy and F1 metric of evaluation data. Using one vs. rest approach the model is evaluated on held out test data with results shown in table 1.Table 1 includes class 1-5: Progressing/worsening/enlarging, stable/no change, improving/responding, not stated/indeterminate or mixed. Summary:The weighted AUCROC ~ 0.97, F1 ~ 0.85 and accuracy ~ 93%. These metric scores improve by choosing notes with higher class probabilities only. Conclusion:Due to the above 0.9 calculated accuracy this NLP model is successful in replicating the curated results of cancer status and the next steps will be to run this model on a new cohort of patients not yet curated. Cancer Status Metrics Class Prevalence Precision Recall F1 Score AUCROC Accuracy 1 38.9% 88.7% 87.0% 87.9% 96.5% 90.7% 2 24.7% 89.9% 88.2% 89.1% 98.3% 94.6% 3 15.9% 87.6% 90.7% 89.1% 98.7% 96.5% 4 15.7% 77.0% 72.6% 74.7% 94.4% 92.3% 5 4.8% 68.1% 74.4% 71.1% 97.9% 97.1% Weighted Average 85% 86% 85% 97% 93% Citation Format: Ankur Arya, Andrew Niederhausern, Nadia S. Bahadur, Neil J. Shah, Chelsea Nichols, Avijit Chatterjee, John Philip. Detection of status of cancer in radiology notes using artificial intelligence [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4965.

Sequence Labelling with 2 Level Segregation (SL2LS): A framework to extract COVID-19 vaccine adverse drug reactions from Twitter data

Social media data is an opportunistic resource for post-marketing surveillance of COVID-19 (Coronavirus disease of 2019) vaccines. However, the usage of the term Adverse Drug Reaction (ADR) in different contexts, which do not refer to adverse reactions to given vaccines, poses a great challenge for existing ADR extraction techniques. In the present work, a framework titled “Sequence Labelling with 2 Level Segregation” (SL2LS) has been proposed to extract adverse reactions of COVID-19 vaccines from Twitter data. It handles the mentioned challenge by deploying a Bidirectional Long Short-Term Memory (Bi-LSTM) based two-level classification model to segregate the tweets containing ADR terms that represent adverse reactions of vaccines from the tweets that comprise ADR terms which are not referring to adverse reactions of vaccines based on the context of ADR term usage and then employ domain-specific pre-trained Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT) sequence labelling model for extracting vaccine linked ADRs from segregated tweets. The proposed framework has been applied to Covaxin and Covishield COVID-19 vaccine data collected from Twitter and annotated with the help of a domain expert. It achieved the highest precision, recall and F1-score of 95.46%, 93.2% and 94.33%, respectively, in comparison to other state-of-the-art techniques. The validation of detected top 20 frequent ADRs of the two vaccines with official surveillance reports resulted in 19 verified ADRs for Covaxin and all 20 verified ADRs for Covishield and one unverified ADR “Blood Clot” for Covaxin which is a valuable lead for medical experts for further exploration. Further analysis shows that the two vaccines share some common ADRs viz. “Discomfort”, “Fever”, “Pain”, “Headache” and “Fatigue” and “Discomfort” and “Fever” are the most frequent ADRs associated with Covaxin and Covishield vaccines, respectively.

Investigating the Impact of Prompt Engineering on the Performance of Large Language Models for Standardizing Obstetric Diagnosis Text: Comparative Study.

The accumulation of vast electronic medical records (EMRs) through medical informatization creates significant research value, particularly in obstetrics. Diagnostic standardization across different health care institutions and regions is vital for medical data analysis. Large language models (LLMs) have been extensively used for various medical tasks. Prompt engineering is key to use LLMs effectively. This study aims to evaluate and compare the performance of LLMs with various prompt engineering techniques on the task of standardizing obstetric diagnostic terminology using real-world obstetric data. The paper describes a 4-step approach used for mapping diagnoses in electronic medical records to the International Classification of Diseases, 10th revision, observation domain. First, similarity measures were used for mapping the diagnoses. Second, candidate mapping terms were collected based on similarity scores above a threshold, to be used as the training data set. For generating optimal mapping terms, we used two LLMs (ChatGLM2 and Qwen-14B-Chat [QWEN]) for zero-shot learning in step 3. Finally, a performance comparison was conducted by using 3 pretrained bidirectional encoder representations from transformers (BERTs), including BERT, whole word masking BERT, and momentum contrastive learning with BERT (MC-BERT), for unsupervised optimal mapping term generation in the fourth step. LLMs and BERT demonstrated comparable performance at their respective optimal levels. LLMs showed clear advantages in terms of performance and efficiency in unsupervised settings. Interestingly, the performance of the LLMs varied significantly across different prompt engineering setups. For instance, when applying the self-consistency approach in QWEN, the F1-score improved by 5%, with precision increasing by 7.9%, outperforming the zero-shot method. Likewise, ChatGLM2 delivered similar rates of accurately generated responses. During the analysis, the BERT series served as a comparative model with comparable results. Among the 3 models, MC-BERT demonstrated the highest level of performance. However, the differences among the versions of BERT in this study were relatively insignificant. After applying LLMs to standardize diagnoses and designing 4 different prompts, we compared the results to those generated by the BERT model. Our findings indicate that QWEN prompts largely outperformed the other prompts, with precision comparable to that of the BERT model. These results demonstrate the potential of unsupervised approaches in improving the efficiency of aligning diagnostic terms in daily research and uncovering hidden information values in patient data.

FakeStack: Hierarchical Tri-BERT-CNN-LSTM stacked model for effective fake news detection.

False news articles pose a serious challenge in today's information landscape, impacting public opinion and decision-making. Efforts to counter this issue have led to research in deep learning and machine learning methods. However, a gap exists in effectively using contextual cues and skip connections within models, limiting the development of comprehensive detection systems that harness contextual information and vital data propagation. Thus, we propose a model of deep learning, FakeStack, in order to identify bogus news accurately. The model combines the power of pre-trained Bidirectional Encoder Representation of Transformers (BERT) embeddings with a deep Convolutional Neural Network (CNN) having skip convolution block and Long Short-Term Memory (LSTM). The model has been trained and tested on English fake news dataset, and various performance metrics were employed to assess its effectiveness. The results showcase the exceptional performance of FakeStack, achieving an accuracy of 99.74%, precision of 99.67%, recall of 99.80%, and F1-score of 99.74%. Our model's performance was extended to two additional datasets. For the LIAR dataset, our accuracy reached 75.58%, while the WELFake dataset showcased an impressive accuracy of 98.25%. Comparative analysis with other baseline models, including CNN, BERT-CNN, and BERT-LSTM, further highlights the superiority of FakeStack, surpassing all models evaluated. This study underscores the potential of advanced techniques in combating the spread of false news and ensuring the dissemination of reliable information.

MetaAc4C: A multi-module deep learning framework for accurate prediction of N4-acetylcytidine sites based on pre-trained bidirectional encoder representation and generative adversarial networks

MotivationN4-acetylcytidine (ac4C) is a highly conserved RNA modification that plays a crucial role in various biological processes. Accurately identifying ac4C sites is of paramount importance for gaining a deeper understanding of their regulatory mechanisms. Nevertheless, the existing experimental techniques for ac4C site identification are characterized by limitations in terms of cost-effectiveness, while the performance of current computational methods in accurately identifying ac4C sites requires further enhancement. ResultsIn this paper, we present MetaAc4C, an advanced deep learning model that leverages pre-trained bidirectional encoder representations from transformers (BERT). The model is based on a bi-directional long short-term memory network (BLSTM) architecture, incorporating attention mechanism and residual connection. To address the issue of data imbalance, we adapt generative adversarial networks to generate synthetic feature samples. On the independent test set, MetaAc4C surpasses the current state-of-the-art ac4C prediction model, exhibiting improvements in terms of ACC, MCC, and AUROC by 2.36%, 4.76%, and 3.11%, respectively, on the unbalanced dataset. When evaluated on the balanced dataset, MetaAc4C achieves improvements in ACC, MCC, and AUROC by 2.6%, 5.11%, and 1.01%, respectively. Notably, our approach of utilizing WGAN-GP augmented training RNA samples demonstrates even superior performance compared to the SMOTE oversampling method.

Graph neural network-based similarity relationship construction model for geospatial services

ABSTRACT During the development of service-based software systems, Geospatial Service (GS) replacement is often performed, which requires the discovery of functionally similar services in service registries to replace failed services. Compared to real-time similarity computations, direct extraction of similar services from constructed similarity relationships can yield higher replacement efficiency. However, missing and inconsistent service-registry information impedes accurate similarity relationship construction. Here, we propose a Graph Neural Network (GNN)-based model for GS Similarity Relationship construction considering service descriptions and tags, which is named GSSR-GNN. As the sparsity of the service similarity relationship graph constructed based on labeled samples limits the information propagation ability, a graph augmentation method for similarity relationship construction among second-order neighbors is proposed. Considering the differences in the semantic-information feature distributions, such as the service descriptions and tags, a feed-forward neural network-based fusion method is designed to embed them into the same vector space. Pre-trained Bidirectional Encoder Representations from Transformers (BERT) and WordNet models are introduced to enhance the service-representation expressiveness. When an enhanced service representation is input to the GNN, the similarity is calculated and the service similarity relationship is obtained. Experimental results show that the proposed model constructs service similarity relationships with high precision, thus improving the service replacement efficiency and reducing the computational cost of service registry during service replacement.

Artificial cheerleading in IEO: Marketing campaign or pump and dump scheme

Initial coin offerings (ICOs) have been vulnerable to pump-and-dump schemes, where fraudsters spread false information on social media to inflate coin prices. However, initial exchange offerings (IEOs) are considered more reliable as exchanges conduct due diligence on projects. This study investigates the effectiveness of IEOs in mitigating pump-and-dump schemes by analyzing over 11,000 tweets posted by different accounts from 54 IEO projects one month prior to the IEO. We select 22 projects with at least 1000 tweets and use topic modeling to perform a descriptive analysis. We use botometer to produce a bot score for each tweet account and a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model for sentiment analysis. We introduce penalized sentiment scores and interaction term and use an OLS regression model to identify the relationship between bot scores, sentiment values, and post-IEO performance. Our research finds that bot score is negatively associated with 30-day return and is positively associated with 30-day volatility, indicating the presence of pump-and-dump schemes in IEOs. We suggest that investors, exchanges, and regulators should take steps to assess the risks associated with IEOs and design appropriate interventions to prevent market manipulation.

A Novel Paradigm for Sentiment Analysis on COVID-19 Tweets with Transfer Learning Based Fine-Tuned BERT

The rapid escalation in global COVID-19 cases has engendered profound emotions of fear, agitation, and despondency within society. It is evident from COVID-19-related tweets that spark panic and elevate stress among individuals. Analyzing the sentiment expressed in online comments aids various stakeholders in monitoring the situation. This research aims to improve the performance of pre-trained bidirectional encoder representations from transformers (BERT) by employing transfer learning (TL) and fine hyper-parameter tuning (FT). The model is applied to three distinct COVID-19-related datasets, and each of the datasets belongs to a different class. The evaluation of the model’s performance involves six different machine learning (ML) classification models. This model is trained and evaluated using metrics such as accuracy, precision, recall, and F1-score. Heat maps are generated for each model to visualize the results. The performance of the model demonstrates accuracies of 83%, 97%, and 98% for Class-5, Class-3, and binary classifications, respectively.

A Robust Drug-Target Interaction Prediction Framework with Capsule Network and Transfer Learning.

Drug-target interactions (DTIs) are considered a crucial component of drug design and drug discovery. To date, many computational methods were developed for drug-target interactions, but they are insufficiently informative for accurately predicting DTIs due to the lack of experimentally verified negative datasets, inaccurate molecular feature representation, and ineffective DTI classifiers. Therefore, we address the limitations of randomly selecting negative DTI data from unknown drug-target pairs by establishing two experimentally validated datasets and propose a capsule network-based framework called CapBM-DTI to capture hierarchical relationships of drugs and targets, which adopts pre-trained bidirectional encoder representations from transformers (BERT) for contextual sequence feature extraction from target proteins through transfer learning and the message-passing neural network (MPNN) for the 2-D graph feature extraction of compounds to accurately and robustly identify drug-target interactions. We compared the performance of CapBM-DTI with state-of-the-art methods using four experimentally validated DTI datasets of different sizes, including human (Homo sapiens) and worm (Caenorhabditis elegans) species datasets, as well as three subsets (new compounds, new proteins, and new pairs). Our results demonstrate that the proposed model achieved robust performance and powerful generalization ability in all experiments. The case study on treating COVID-19 demonstrates the applicability of the model in virtual screening.

A prompt model with combined semantic refinement for aspect sentiment analysis

Recently, pre-trained language models (PLMs), especially pre-trained bidirectional encoder representations from transformers (BERT), have improved the performance of aspect-based sentiment analysis (ABSA) tasks to some extent. However, due to the imbalance of training data in different polarities, the following shortcomings remain in PLM-based ABSA methods: (1) for small corpus scenarios with polarized emotions, an unbalanced performance problem exists; and (2) for delicate and obscure scenes dominated by neutral emotions, PLM-based performance gains are limited. To address these shortcomings, we use BERT as an instance of PLMs to propose a general-purpose prompt model with combined semantic refinement for ABSA. First, we utilize a BERT without fine-tuning to automatically induce prompts for various ABSA datasets to enhance the adaptability of the model to different application scenarios. We then leverage multi-prompt learning to propose a data augmentation method to address the imbalance of training data in different polarities. Moreover, to further deepen the model's understanding and analysis of reviews with prompts, we also propose an improved BERT semantic refinement method that combines global semantic refinement and local semantic extraction. Experiments on five public datasets show that compared with existing methods, our macro-average F1 improvement is over 10% on polarized small datasets and over 7% on an emotionally delicate and obscure dataset.

AD-BERT: Using pre-trained language model to predict the progression from mild cognitive impairment to Alzheimer's disease

ObjectiveWe develop a deep learning framework based on the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model using unstructured clinical notes from electronic health records (EHRs) to predict the risk of disease progression from Mild Cognitive Impairment (MCI) to Alzheimer's Disease (AD). MethodsWe identified 3657 patients diagnosed with MCI together with their progress notes from Northwestern Medicine Enterprise Data Warehouse (NMEDW) between 2000 and 2020. The progress notes no later than the first MCI diagnosis were used for the prediction. We first preprocessed the notes by deidentification, cleaning and splitting into sections, and then pre-trained a BERT model for AD (named AD-BERT) based on the publicly available Bio+Clinical BERT on the preprocessed notes. All sections of a patient were embedded into a vector representation by AD-BERT and then combined by global MaxPooling and a fully connected network to compute the probability of MCI-to-AD progression. For validation, we conducted a similar set of experiments on 2563 MCI patients identified at Weill Cornell Medicine (WCM) during the same timeframe. ResultsCompared with the 7 baseline models, the AD-BERT model achieved the best performance on both datasets, with Area Under receiver operating characteristic Curve (AUC) of 0.849 and F1 score of 0.440 on NMEDW dataset, and AUC of 0.883 and F1 score of 0.680 on WCM dataset. ConclusionThe use of EHRs for AD-related research is promising, and AD-BERT shows superior predictive performance in modeling MCI-to-AD progression prediction. Our study demonstrates the utility of pre-trained language models and clinical notes in predicting MCI-to-AD progression, which could have important implications for improving early detection and intervention for AD.

Improving question answering performance using knowledge distillation and active learning

Contemporary question answering (QA) systems, including Transformer-based architectures, suffer from increasing computational and model complexity which render them inefficient for real-world applications with limited resources. Furthermore, training or even fine-tuning such models requires a vast amount of labeled data which is often not available for the task at hand. In this manuscript, we conduct a comprehensive analysis of the mentioned challenges and introduce suitable countermeasures. We propose a novel knowledge distillation (KD) approach to reduce the parameter and model complexity of a pre-trained bidirectional encoder representations from transformer (BERT) system and utilize multiple active learning (AL) strategies for immense reduction in annotation efforts. We show the efficacy of our approach by comparing it with four state-of-the-art (SOTA) Transformers-based systems, namely KroneckerBERT, EfficientBERT, TinyBERT, and DistilBERT. Specifically, we outperform KroneckerBERT21 and EfficientBERTTINY by 4.5 and 0.4 percentage points in EM, despite having 75.0% and 86.2% fewer parameters, respectively. Additionally, our approach achieves comparable performance to 6-layer TinyBERT and DistilBERT while using only 2% of their total trainable parameters. Besides, by the integration of our AL approaches into the BERT framework, we show that SOTA results on the QA datasets can be achieved when we only use 40% of the training data. Overall, all results demonstrate the effectiveness of our approaches in achieving SOTA performance, while extremely reducing the number of parameters and labeling efforts. Finally, we make our code publicly available at https://github.com/mirbostani/QA-KD-AL.

M6A-BERT-Stacking: A Tissue-Specific Predictor for Identifying RNA N6-Methyladenosine Sites Based on BERT and Stacking Strategy

As the most abundant RNA methylation modification, N6-methyladenosine (m6A) could regulate asymmetric and symmetric division of hematopoietic stem cells and play an important role in various diseases. Therefore, the precise identification of m6A sites around the genomes of different species is a critical step to further revealing their biological functions and influence on these diseases. However, the traditional wet-lab experimental methods for identifying m6A sites are often laborious and expensive. In this study, we proposed an ensemble deep learning model called m6A-BERT-Stacking, a powerful predictor for the detection of m6A sites in various tissues of three species. First, we utilized two encoding methods, i.e., di ribonucleotide index of RNA (DiNUCindex_RNA) and k-mer word segmentation, to extract RNA sequence features. Second, two encoding matrices together with the original sequences were respectively input into three different deep learning models in parallel to train three sub-models, namely residual networks with convolutional block attention module (Resnet-CBAM), bidirectional long short-term memory with attention (BiLSTM-Attention), and pre-trained bidirectional encoder representations from transformers model for DNA-language (DNABERT). Finally, the outputs of all sub-models were ensembled based on the stacking strategy to obtain the final prediction of m6A sites through the fully connected layer. The experimental results demonstrated that m6A-BERT-Stacking outperformed most of the existing methods based on the same independent datasets.

Twitter-based classification for integrated source data of weather observations

<span lang="EN-US">Meteorology and weather forecasting are crucial for predicting future climate conditions. Forecasts can be helpful when they provide information that can assist people in making better decisions. People today use big data to analyze social media information accurately, including those who rely on the weather forecast. Recent years have seen the widespread use of machine learning and deep learning for managing messages on social media sites like Twitter. In this study, authors analyzed weather-related text in Indonesia based on the searches made on Twitter. A total of three machine learning algorithms were examined: support vector machine (SVM), multinomial logistic regression (MLR), and multinomial Naive Bayes (MNB), as well as the pretrained bidirectional encoder representations of transformers (BERT), which was fine-tuned over multiple layers to ensure effective classification. The accuracy of the BERT model, calculated using the F1-score of 99%, was higher than that of any other machine learning method. Those results have been incorporated into a web-based weather information system. The classification result was mapped using Esri Maps application programming interface (API) based on the geolocation of the data.</span>

Fine-tuning BERT for automatic ADME semantic labeling in FDA drug labeling to enhance product-specific guidance assessment

Product-specific guidances (PSGs) recommended by the United States Food and Drug Administration (FDA) are instrumental to promote and guide generic drug product development. To assess a PSG, the FDA assessor needs to take extensive time and effort to manually retrieve supportive drug information of absorption, distribution, metabolism, and excretion (ADME) from the reference listed drug labeling. In this work, we leveraged the state-of-the-art pre-trained language models to automatically label the ADME paragraphs in the pharmacokinetics section from the FDA-approved drug labeling to facilitate PSG assessment. We applied a transfer learning approach by fine-tuning the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to develop a novel application of ADME semantic labeling, which can automatically retrieve ADME paragraphs from drug labeling instead of manual work. We demonstrate that fine-tuning the pre-trained BERT model can outperform conventional machine learning techniques, achieving up to 12.5% absolute F1 improvement. To our knowledge, we were the first to successfully apply BERT to solve the ADME semantic labeling task. We further assessed the relative contribution of pre-training and fine-tuning to the overall performance of the BERT model in the ADME semantic labeling task using a series of analysis methods, such as attention similarity and layer-based ablations. Our analysis revealed that the information learned via fine-tuning is focused on task-specific knowledge in the top layers of the BERT, whereas the benefit from the pre-trained BERT model is from the bottom layers.

AI Assisted Attention Mechanism for Hybrid Neural Model to Assess Online Attitudes About COVID-19.

COVID-19 is a novel virus that presents challenges due to a lack of consistent and in-depth research. The news of the COVID-19 spreads across the globe, resulting in a flood of posts on social media sites. Apart from health, social, and economic disturbances brought by the COVID-19 pandemic, another important consequence involves public mental health crises which is of greater concern. Data related to COVID-19 is a valuable asset for researchers in understanding people's feelings related to the pandemic. It is thus important to extract the early information evolving public sentiments on social platforms during the outbreak of COVID-19. The objective of this studyis to look at people's perceptions of the COVID-19pandemicwho interact with each other and share tweets on the Twitter platform. COVIDSenti, a large-scale benchmark dataset comprising 90,000 COVID-19 tweets collected from February to March 2020,during the initial phases of the outbreak served as the foundation for our experiments. A pre-trained bidirectional encoder representations from transformers (BERT) model is fine-tuned and embeddings generated are combined with two long short-term memory networks to propose the residual encoder transformation network model. The proposed model is used for multiclass text classification on a large dataset labeled as positive, negative, and neutral. The experimental outcomes validate that: (1) the proposed model is the best performing model, with 98% accuracy and 96% F1-score; (2) It also outperforms conventional machine learning algorithms and different variants of BERT, and (3) the approach achieves better results as compared to state-of-the-art on different benchmark datasets.

Predicting medical specialty from text based on a domain-specific pre-trained BERT

BackgroundOwing to the prevalence of the coronavirus disease (COVID-19), coping with clinical issues at the individual level has become important to the healthcare system. Accordingly, precise initiation of treatment after a hospital visit is required for expedited processes and effective diagnoses of outpatients. To achieve this, artificial intelligence in medical natural language processing (NLP), such as a healthcare chatbot or a clinical decision support system, can be suitable tools for an advanced clinical system. Furthermore, support for decisions on the medical specialty from the initial visit can be helpful. Materials and methodsIn this study, we propose a medical specialty prediction model from patient-side medical question text based on pre-trained bidirectional encoder representations from transformers (BERT). The dataset comprised pairs of medical question texts and labeled specialties scraped from a website for the medical question-and-answer service. The model was fine-tuned for predicting the required medical specialty labels among 27 labels from medical question texts. To demonstrate the feasibility, we conducted experiments on a real-world dataset and elaborately evaluated the predictive performance compared with four deep learning NLP models through cross-validation and test set evaluation. ResultsThe proposed model showed improved performance compared with competitive models in terms of overall specialties. In addition, we demonstrate the usefulness of the proposed model by performing case studies for visualization applications. ConclusionThe proposed model can benefit hospital patient management and reasonable recommendations for specialties for patients.