Bidirectional Encoder Representations From Transformers Model Research Articles

BERT models for Brazilian Portuguese: Pretraining, evaluation and tokenization analysis

Recent advances in language representation using neural networks have made it viable to transfer the learned internal states of large pretrained language models (LMs) to downstream natural language processing (NLP) tasks. This transfer learning approach improves the overall performance on many tasks and is highly beneficial when labeled data is scarce, making pretrained LMs valuable resources specially for languages with few annotated training examples. In this work, we train BERT (Bidirectional Encoder Representations from Transformers) models for Brazilian Portuguese, which we nickname BERTimbau. We evaluate our models on three downstream NLP tasks: sentence textual similarity, recognizing textual entailment, and named entity recognition. Our models improve the state-of-the-art in all of these tasks, outperforming Multilingual BERT and confirming the effectiveness of large pretrained LMs for Portuguese. We release our models to the community hoping to provide strong baselines for future NLP research: https://github.com/neuralmind-ai/portuguese-bert.

A Natural Language Processing Model for COVID-19 Detection Based on Dutch General Practice Electronic Health Records by Using Bidirectional Encoder Representations From Transformers: Development and Validation Study.

Natural language processing (NLP) models such as bidirectional encoder representations from transformers (BERT) hold promise in revolutionizing disease identification from electronic health records (EHRs) by potentially enhancing efficiency and accuracy. However, their practical application in practice settings demands a comprehensive and multidisciplinary approach to development and validation. The COVID-19 pandemic highlighted challenges in disease identification due to limited testing availability and challenges in handling unstructured data. In the Netherlands, where general practitioners (GPs) serve as the first point of contact for health care, EHRs generated by these primary care providers contain a wealth of potentially valuable information. Nonetheless, the unstructured nature of free-text entries in EHRs poses challenges in identifying trends, detecting disease outbreaks, or accurately pinpointing COVID-19 cases. This study aims to develop and validate a BERT model for detecting COVID-19 consultations in general practice EHRs in the Netherlands. The BERT model was initially pretrained on Dutch language data and fine-tuned using a comprehensive EHR data set comprising confirmed COVID-19 GP consultations and non-COVID-19-related consultations. The data set was partitioned into a training and development set, and the model's performance was evaluated on an independent test set that served as the primary measure of its effectiveness in COVID-19 detection. To validate the final model, its performance was assessed through 3 approaches. First, external validation was applied on an EHR data set from a different geographic region in the Netherlands. Second, validation was conducted using results of polymerase chain reaction (PCR) test data obtained from municipal health services. Lastly, correlation between predicted outcomes and COVID-19-related hospitalizations in the Netherlands was assessed, encompassing the period around the outbreak of the pandemic in the Netherlands, that is, the period before widespread testing. The model development used 300,359 GP consultations. We developed a highly accurate model for COVID-19 consultations (accuracy 0.97, F1-score 0.90, precision 0.85, recall 0.85, specificity 0.99). External validations showed comparable high performance. Validation on PCR test data showed high recall but low precision and specificity. Validation using hospital data showed significant correlation between COVID-19 predictions of the model and COVID-19-related hospitalizations (F1-score 96.8; P<.001; R2=0.69). Most importantly, the model was able to predict COVID-19 cases weeks before the first confirmed case in the Netherlands. The developed BERT model was able to accurately identify COVID-19 cases among GP consultations even preceding confirmed cases. The validated efficacy of our BERT model highlights the potential of NLP models to identify disease outbreaks early, exemplifying the power of multidisciplinary efforts in harnessing technology for disease identification. Moreover, the implications of this study extend beyond COVID-19 and offer a blueprint for the early recognition of various illnesses, revealing that such models could revolutionize disease surveillance.

Multi-label text classification of Indonesian customer reviews using bidirectional encoder representations from transformers language model

&lt;p&gt;&lt;span lang="EN-US"&gt;Customer review is a critical resource to support the decision-making process in various industries. To understand how customers perceived each aspect of the product, we can first identify all aspects discussed in the customer reviews by performing multi-label text classification. In this work, we want to know the effectiveness of our two proposed strategies using bidirectional encoder representations from transformers (BERT) language model that was&lt;br /&gt; pre-trained on the Indonesian language, referred to as IndoBERT, to perform multi-label text classification. First, IndoBERT is used as feature representation to be combined with convolutional neural network-extreme gradient boosting (CNN-XGBoost). Second, IndoBERT is used both as the feature representation as well as the classifier to directly solve the classification task. Additional analysis is performed to compare our results with those using multilingual BERT model. According to our experimental results, our first model using IndoBERT as feature representation shows significant performance over some baselines. Our second model using IndoBERT as both feature representation and classifier can significantly enhance the effectiveness of our first model. In summary, our proposed models can improve the effectiveness of the baseline using Word2Vec-CNN-XGBoost by 19.19% and 6.17%, in terms of accuracy and F-1 score, respectively.&lt;/span&gt;&lt;/p&gt;

Transformer-based approach to variable typing

The upsurge of multifarious endeavors across scientific fields propelled Big Data in the scientific domain. Despite the advancements in management systems, researchers find that mathematical knowledge remains one of the most challenging to manage due to the latter's inherent heterogeneity. One novel recourse being explored is variable typing where current works remain preliminary and, thus, provide a wide room for contribution. In this study, a primordial attempt to implement the end-to-end Entity Recognition (ER) and Relation Extraction (RE) approach to variable typing was made using the BERT (Bidirectional Encoder Representations from Transformers) model. A micro-dataset was developed for this process. According to our findings, the ER model and RE model, respectively, have Precision of 0.8142 and 0.4919, Recall of 0.7816 and 0.6030, and F1-Scores of 0.7975 and 0.5418. Despite the limited dataset, the models performed at par with values in the literature. This work also discusses the factors affecting this BERT-based approach, giving rise to suggestions for future implementations.

Identifying the Question Similarity of Regulatory Documents in the Pharmaceutical Industry by Using the Recognizing Question Entailment System: Evaluation Study.

The regulatory affairs (RA) division in a pharmaceutical establishment is the point of contact between regulatory authorities and pharmaceutical companies. They are delegated the crucial and strenuous task of extracting and summarizing relevant information in the most meticulous manner from various search systems. An artificial intelligence (AI)-based intelligent search system that can significantly bring down the manual efforts in the existing processes of the RA department while maintaining and improving the quality of final outcomes is desirable. We proposed a "frequently asked questions" component and its utility in an AI-based intelligent search system in this paper. The scenario is further complicated by the lack of publicly available relevant data sets in the RA domain to train the machine learning models that can facilitate cognitive search systems for regulatory authorities. In this study, we aimed to use AI-based intelligent computational models to automatically recognize semantically similar question pairs in the RA domain and evaluate the Recognizing Question Entailment-based system. We used transfer learning techniques and experimented with transformer-based models pretrained on corpora collected from different resources, such as Bidirectional Encoder Representations from Transformers (BERT), Clinical BERT, BioBERT, and BlueBERT. We used a manually labeled data set that contained 150 question pairs in the pharmaceutical regulatory domain to evaluate the performance of our model. The Clinical BERT model performed better than other domain-specific BERT-based models in identifying question similarity from the RA domain. The BERT model had the best ability to learn domain-specific knowledge with transfer learning, which reached the best performance when fine-tuned with sufficient clinical domain question pairs. The top-performing model achieved an accuracy of 90.66% on the test set. This study demonstrates the possibility of using pretrained language models to recognize question similarity in the pharmaceutical regulatory domain. Transformer-based models that are pretrained on clinical notes perform better than models pretrained on biomedical text in recognizing the question's semantic similarity in this domain. We also discuss the challenges of using data augmentation techniques to address the lack of relevant data in this domain. The results of our experiment indicated that increasing the number of training samples using back translation and entity replacement did not enhance the model's performance. This lack of improvement may be attributed to the intricate and specialized nature of texts in the regulatory domain. Our work provides the foundation for further studies that apply state-of-the-art linguistic models to regulatory documents in the pharmaceutical industry.

Using Social Media to Help Understand Patient-Reported Health Outcomes of Post-COVID-19 Condition: Natural Language Processing Approach.

While scientific knowledge of post-COVID-19 condition (PCC) is growing, there remains significant uncertainty in the definition of the disease, its expected clinical course, and its impact on daily functioning. Social media platforms can generate valuable insights into patient-reported health outcomes as the content is produced at high resolution by patients and caregivers, representing experiences that may be unavailable to most clinicians. In this study, we aimed to determine the validity and effectiveness of advanced natural language processing approaches built to derive insight into PCC-related patient-reported health outcomes from social media platforms Twitter and Reddit. We extracted PCC-related terms, including symptoms and conditions, and measured their occurrence frequency. We compared the outputs with human annotations and clinical outcomes and tracked symptom and condition term occurrences over time and locations to explore the pipeline's potential as a surveillance tool. We used bidirectional encoder representations from transformers (BERT) models to extract and normalize PCC symptom and condition terms from English posts on Twitter and Reddit. We compared 2 named entity recognition models and implemented a 2-step normalization task to map extracted terms to unique concepts in standardized terminology. The normalization steps were done using a semantic search approach with BERT biencoders. We evaluated the effectiveness of BERT models in extracting the terms using a human-annotated corpus and a proximity-based score. We also compared the validity and reliability of the extracted and normalized terms to a web-based survey with more than 3000 participants from several countries. UmlsBERT-Clinical had the highest accuracy in predicting entities closest to those extracted by human annotators. Based on our findings, the top 3 most commonly occurring groups of PCC symptom and condition terms were systemic (such as fatigue), neuropsychiatric (such as anxiety and brain fog), and respiratory (such as shortness of breath). In addition, we also found novel symptom and condition terms that had not been categorized in previous studies, such as infection and pain. Regarding the co-occurring symptoms, the pair of fatigue and headaches was among the most co-occurring term pairs across both platforms. Based on the temporal analysis, the neuropsychiatric terms were the most prevalent, followed by the systemic category, on both social media platforms. Our spatial analysis concluded that 42% (10,938/26,247) of the analyzed terms included location information, with the majority coming from the United States, United Kingdom, and Canada. The outcome of our social media-derived pipeline is comparable with the results of peer-reviewed articles relevant to PCC symptoms. Overall, this study provides unique insights into patient-reported health outcomes of PCC and valuable information about the patient's journey that can help health care providers anticipate future needs. RR2-10.1101/2022.12.14.22283419.

Prediction of multiple types of RNA modifications via biological language model.

It has been demonstrated that RNA modifications play essential roles in multiple biological processes. Accurate identification of RNA modifications in the transcriptome is critical for providing insights into the biological functions and mechanisms. Many tools have been developed for predicting RNA modifications at single-base resolution, which employ conventional feature engineering methods that focus on feature design and feature selection processes that require extensive biological expertise and may introduce redundant information. With the rapid development of artificial intelligence technologies, end-to-end methods are favorably received by researchers. Nevertheless, each well-trained model is only suitable for a specific RNA methylation modification type for nearly all of these approaches. In this study, we present MRM-BERT by feeding task-specific sequences into the powerful BERT (Bidirectional Encoder Representations from Transformers) model and implementing fine-tuning, which exhibits competitive performance to the state-of-the-art methods. MRM-BERT avoids repeated de novo training of the model and can predict multiple RNA modifications such as pseudouridine, m6A, m5C, and m1A in Mus musculus, Arabidopsis thaliana, and Saccharomyces cerevisiae. In addition, we analyse the attention heads to provide high attention regions for the prediction, and conduct saturated in silico mutagenesis of the input sequences to discover potential changes of RNA modifications, which can better assist researchers in their follow-up research. MRM-BERT is freely accessible at http://csbio.njust.edu.cn/bioinf/mrmbert/.

Automated decision classification model for tax appeals commission in Morocco using latent dirichlet allocation

&lt;p&gt;This research paper focuses on extracting and classifying information from the Moroccan National Tax Appeals Commission, which is presently nonexistent in the country's legal and tax landscape. This study examines 201 decisions selected from a pool of 562, released between 1999 and 2018, pertaining to corporate tax and involving 550 disputes spanning various corporate tax classifications. The paper aims to propose latent dirichlet allocation (LDA) for topic modeling and compare it with our previous results obtained from the bidirectional encoder representations from transformers (BERT) model. The findings suggest that the rulings can be classified into two primary classifications: those that uphold or reject the tax administration's position. The proposed model shows a good performance, achieving a precision of 9.25% and an accuracy of 9.51%. This highlights the effectiveness of both LDA and BERT models for understanding and classifying topics in tax decision analysis.&lt;/p&gt;

Examining the relationship between peer feedback classified by deep learning and online learning burnout

Students are prone to experiencing learning burnout while engaged in online learning due to the lack of interaction. However, peer feedback can play an important role in alleviating learning burnout. This study uses a combination of deep learning (DL) and statistical analysis to investigate the relationship between online peer feedback and learning burnout. Several DL algorithms, including Bidirectional Encoder Representations from Transformers (BERT), Bi-directional Long Short-Term Memory (Bi-LSTM), and BERT-Bi-LSTM, are compared to classify peer comments in terms of cognitive content and affective state. Moreover, by using the feedback classified by DL and survey data from 116 participants, multiple linear regression (MLR) analysis is employed to explore the relationships between online learning burnout and feedback messages. The results show that the BERT model achieves the best performance in terms of classification performance and agreement with manual coding. Further, we find that student burnout is significantly influenced by peer feedback. Individuals who receive more suggestive feedback experience a greater reduction in emotional exhaustion. However, when receiving more reinforcing feedback without guidance, learners' behavior tends to deteriorate. From an affective perspective, the more positive feedback the learners receive, the more they tend to exhibit higher levels of emotional exhaustion and a diminished sense of achievement. And when receiving more negative feedback, learners tend to have a worse emotional experience and demonstrate poorer self-learning behavior. This study contributes to our understanding of the application of DL in peer assessment and the impact of received peer feedback on preventing and alleviating learning burnout.

Analyzing Sentiments Regarding ChatGPT Using Novel BERT: A Machine Learning Approach

Chatbots are AI-powered programs designed to replicate human conversation. They are capable of performing a wide range of tasks, including answering questions, offering directions, controlling smart home thermostats, and playing music, among other functions. ChatGPT is a popular AI-based chatbot that generates meaningful responses to queries, aiding people in learning. While some individuals support ChatGPT, others view it as a disruptive tool in the field of education. Discussions about this tool can be found across different social media platforms. Analyzing the sentiment of such social media data, which comprises people’s opinions, is crucial for assessing public sentiment regarding the success and shortcomings of such tools. This study performs a sentiment analysis and topic modeling on ChatGPT-based tweets. ChatGPT-based tweets are the author’s extracted tweets from Twitter using ChatGPT hashtags, where users share their reviews and opinions about ChatGPT, providing a reference to the thoughts expressed by users in their tweets. The Latent Dirichlet Allocation (LDA) approach is employed to identify the most frequently discussed topics in relation to ChatGPT tweets. For the sentiment analysis, a deep transformer-based Bidirectional Encoder Representations from Transformers (BERT) model with three dense layers of neural networks is proposed. Additionally, machine and deep learning models with fine-tuned parameters are utilized for a comparative analysis. Experimental results demonstrate the superior performance of the proposed BERT model, achieving an accuracy of 96.49%.

Identifying Risk Factors Associated With Lower Back Pain in Electronic Medical Record Free Text: Deep Learning Approach Using Clinical Note Annotations.

Lower back pain is a common weakening condition that affects a large population. It is a leading cause of disability and lost productivity, and the associated medical costs and lost wages place a substantial burden on individuals and society. Recent advances in artificial intelligence and natural language processing have opened new opportunities for the identification and management of risk factors for lower back pain. In this paper, we propose and train a deep learning model on a data set of clinical notes that have been annotated with relevant risk factors, and we evaluate the model's performance in identifying risk factors in new clinical notes. The primary objective is to develop a novel deep learning approach to detect risk factors for underlying disease in patients presenting with lower back pain in clinical encounter notes. The secondary objective is to propose solutions to potential challenges of using deep learning and natural language processing techniques for identifying risk factors in electronic medical record free text and make practical recommendations for future research in this area. We manually annotated clinical notes for the presence of six risk factors for severe underlying disease in patients presenting with lower back pain. Data were highly imbalanced, with only 12% (n=296) of the annotated notes having at least one risk factor. To address imbalanced data, a combination of semantic textual similarity and regular expressions was used to further capture notes for annotation. Further analysis was conducted to study the impact of downsampling, binary formulation of multi-label classification, and unsupervised pretraining on classification performance. Of 2749 labeled clinical notes, 347 exhibited at least one risk factor, while 2402 exhibited none. The initial analysis shows that downsampling the training set to equalize the ratio of clinical notes with and without risk factors improved the macro-area under the receiver operating characteristic curve (AUROC) by 2%. The Bidirectional Encoder Representations from Transformers (BERT) model improved the macro-AUROC by 15% over the traditional machine learning baseline. In experiment 2, the proposed BERT-convolutional neural network (CNN) model for longer texts improved (4% macro-AUROC) over the BERT baseline, and the multitask models are more stable for minority classes. In experiment 3, domain adaptation of BERTCNN using masked language modeling improved the macro-AUROC by 2%. Primary care clinical notes are likely to require manipulation to perform meaningful free-text analysis. The application of BERT models for multi-label classification on downsampled annotated clinical notes is useful in detecting risk factors suggesting an indication for imaging for patients with lower back pain.

An Empirical Study on Adaptive Inference for Pretrained Language Model.

Adaptive inference has been proven to improve bidirectional encoder representations from transformers (BERT)'s inference speed with minimal loss of accuracy. However, current work only focuses on the BERT model and lacks exploration of other pretrained language models (PLMs). Therefore, this article conducts an empirical study on the application of adaptive inference mechanism in various PLMs, including generative pretraining (GPT), GCNN, ALBERT, and TinyBERT. This mechanism is verified on both English and Chinese benchmarks, and experimental results demonstrated that it is able to speed up by a wide range from 1 to 10 times if given different speed thresholds. In addition, its application on ALBERT shows that adaptive inference can work with parameter sharing, achieving model compression and acceleration simultaneously, while the application on TinyBERT proves that it can further accelerate the distilled small model. As for the problem that too many labels make adaptive inference invalid, this article also proposes a solution, namely label reduction. Finally, this article open-sources an easy-to-use toolkit called FastPLM to help developers adopt pretrained models with adaptive inference capabilities in their applications.

Information extraction from weakly structured radiological reports with natural language queries

ObjectivesProvide physicians and researchers an efficient way to extract information from weakly structured radiology reports with natural language processing (NLP) machine learning models.MethodsWe evaluate seven different German bidirectional encoder representations from transformers (BERT) models on a dataset of 857,783 unlabeled radiology reports and an annotated reading comprehension dataset in the format of SQuAD 2.0 based on 1223 additional reports.ResultsContinued pre-training of a BERT model on the radiology dataset and a medical online encyclopedia resulted in the most accurate model with an F1-score of 83.97% and an exact match score of 71.63% for answerable questions and 96.01% accuracy in detecting unanswerable questions. Fine-tuning a non-medical model without further pre-training led to the lowest-performing model. The final model proved stable against variation in the formulations of questions and in dealing with questions on topics excluded from the training set.ConclusionsGeneral domain BERT models further pre-trained on radiological data achieve high accuracy in answering questions on radiology reports. We propose to integrate our approach into the workflow of medical practitioners and researchers to extract information from radiology reports.Clinical relevance statementBy reducing the need for manual searches of radiology reports, radiologists’ resources are freed up, which indirectly benefits patients.Key Points• BERT models pre-trained on general domain datasets and radiology reports achieve high accuracy (83.97% F1-score) on question-answering for radiology reports.• The best performing model achieves an F1-score of 83.97% for answerable questions and 96.01% accuracy for questions without an answer.• Additional radiology-specific pretraining of all investigated BERT models improves their performance.Graphical

Automated Recognition of Visual Acuity Measurements in Ophthalmology Clinical Notes Using Deep Learning

Visual acuity (VA) is a critical component of the eye examination but is often only documented in electronic health records (EHRs) as unstructured free-text notes, making it challenging to use in research. This study aimed to improve on existing rule-based algorithms by developing and evaluating deep learning models to perform named entity recognition of different types of VA measurements and their lateralities from free-text ophthalmology notes: VA for each of the right and left eyes, with and without glasses correction, and with and without pinhole. Cross-sectional study. A total of 319 756 clinical notes with documented VA measurements from approximately 90 000 patients were included. The notes were split into train, validation, and test sets. Bidirectional Encoder Representations from Transformers (BERT) models were fine-tuned to identify VA measurements from the progress notes and included BERT models pretrained on biomedical literature (BioBERT), critical care EHR notes (ClinicalBERT), both (BlueBERT), and a lighter version of BERT with 40% fewer parameters (DistilBERT). A baseline rule-based algorithm was created to recognize the same VA entities to compare against BERT models. Model performance was evaluated on a held-out test set using microaveraged precision, recall, and F1 score for all entities. On the human-annotated subset, BlueBERT achieved the best microaveraged F1 score (F1= 0.92), followed by ClinicalBERT (F1= 0.91), DistilBERT (F1= 0.90), BioBERT (F1= 0.84), and the baseline model (F1= 0.83). Common errors included labeling VA in sections outside of the examination portion of the note, difficulties labeling current VA alongside a series of past VAs, and missing nonnumeric VAs. This study demonstrates that deep learning models are capable of identifying VA measurements from free-text ophthalmology notes with high precision and recall, achieving significant performance improvements over a rule-based algorithm. The ability to recognize VA from free-text notes would enable a more detailed characterization of ophthalmology patient cohorts and enhance the development of models to predict ophthalmology outcomes. Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Protecting Children from Harmful Audio Content: Automated Profanity Detection From English Audio in Songs and Social-Media

A novel approach for the automated detection of profanity in English audio songs using machine learning techniques. One of the primary drawbacks of existing systems is only confined to textual data. The proposed method utilizes a combination of feature extraction techniques and machine learning algorithms to identify profanity in audio songs. Specifically, the approach employs the popular feature extraction techniques of Term frequency–inverse document frequency (TF-IDF), Bidirectional Encoder Representations from Transformers (BERT) and Doc2vec to extract relevant features from the audio songs. TF-IDF is used to capture the frequency and importance of each word in the song, while BERT is utilized to extract contextualized representations of words that can capture more nuanced meanings. To capture the semantic meaning of words in audio songs, also explored the use of the Doc2Vec model, which is a neural network-based approach that can extract relevant features from the audio songs. The study utilizes Open Whisper, an open-source machine learning library, to develop and implement the approach. A dataset of English audio songs was used to evaluate the performance of the proposed method. The results showed that both the TF-IDF and BERT models outperformed the Doc2Vec model in terms of accuracy in identifying profanity in English audio songs. The proposed approach has potential applications in identifying profanity in various forms of audio content, including songs, audio clips, social media, reels, and shorts.

Improving attention network to realize joint extraction for the construction of equipment knowledge graph

In the context of promoting autonomous decision, intelligent interaction, and concept recognition of the equipment, constructing an equipment Knowledge Graph (KG) is the key technology to facilitate this. In this article, we propose a method that combines design rules and automatic extraction to construct equipment KG from equipment manuals. We define that equipment KG is scheme-based and depicts scheme in the form of KG triples (a.k.a design rules, scheme layer of KG, or concept KG). Our contributions include designing an initial concept KG for the equipment domain and improving the BERT (Bidirectional Encoder Representations from Transformers) model to jointly extract knowledge from texts to enrich the KG. The BERT model was improved from internal calculations so that joint extraction could be achieved directly without extra additional parts. We applied our model to a standard dataset “SemEval2010 Task 8” and achieved the F1 score of 89.55 which demonstrates its rationality. We also established a dataset called “Equipment Manuals Corpus for KG” based on the concept KG and applied the joint model in the dataset to extract knowledge. The result was visualized in the form of a graph.

AD-BERT: Using pre-trained language model to predict the progression from mild cognitive impairment to Alzheimer's disease

ObjectiveWe develop a deep learning framework based on the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model using unstructured clinical notes from electronic health records (EHRs) to predict the risk of disease progression from Mild Cognitive Impairment (MCI) to Alzheimer's Disease (AD). MethodsWe identified 3657 patients diagnosed with MCI together with their progress notes from Northwestern Medicine Enterprise Data Warehouse (NMEDW) between 2000 and 2020. The progress notes no later than the first MCI diagnosis were used for the prediction. We first preprocessed the notes by deidentification, cleaning and splitting into sections, and then pre-trained a BERT model for AD (named AD-BERT) based on the publicly available Bio+Clinical BERT on the preprocessed notes. All sections of a patient were embedded into a vector representation by AD-BERT and then combined by global MaxPooling and a fully connected network to compute the probability of MCI-to-AD progression. For validation, we conducted a similar set of experiments on 2563 MCI patients identified at Weill Cornell Medicine (WCM) during the same timeframe. ResultsCompared with the 7 baseline models, the AD-BERT model achieved the best performance on both datasets, with Area Under receiver operating characteristic Curve (AUC) of 0.849 and F1 score of 0.440 on NMEDW dataset, and AUC of 0.883 and F1 score of 0.680 on WCM dataset. ConclusionThe use of EHRs for AD-related research is promising, and AD-BERT shows superior predictive performance in modeling MCI-to-AD progression prediction. Our study demonstrates the utility of pre-trained language models and clinical notes in predicting MCI-to-AD progression, which could have important implications for improving early detection and intervention for AD.

CoBERT: A Contextual BERT model for recommending employability profiles of information technology students in unstable developing countries

Unemployment constitutes one of the major problems in developing countries, with factors such as unavailable skills and the proliferation of unskilled workers being cited as main causes. The existing literature has shown that using social networks analysis techniques, employment options for students and graduates can be accurately generated through recommender systems (RS) to optimize the chances of employment. Therefore, this study proposes a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model named Contextual BERT (CoBERT) for item-centered content-based (CB) filtering RS. The model uses socio-political background, graduate-employer relationship and graduate academic competencies of information technology (IT) students as contextual factors to recommend employability profiles of IT students in unstable developing countries (UDCs) such as the Democratic Republic of Congo (DRC). The performance of the proposed model is verified using four metrics. To evaluate the online performance of the model, the metrics of Diversity and Novelty are used. The metrics of normalized discounted cumulative gain (nDCG) and mean average precision (MAP) are applied to evaluate the model offline. The proposed model shows a high nDCG value at Top-N=4, equaling to 0.99. Validation statistics of 0.5 MAP score and significant t-statistics test values for Diversity and Novelty indicate that the proposed model can be generalized in the DRC and other similar UDCs. The RS model proposed in the current study contributes to the literature on employability models by advancing beyond traditional approaches such as fuzzy logic (FL), and also by taking graduate skills into account. The proposed model is a novel employability recommendation technique that provides highly precise and contextual recommendations. The output from this study contributes a theoretical foundation for implementing RSs in education and the industry.

MTBERT-Attention: An Explainable BERT Model based on Multi-Task Learning for Cognitive Text Classification

In recent years, there has been a lot of focus on Bloom’s taxonomy-based classification of E-Learning materials. Researchers employ different methods and features. In our previous works, we have dealt with this problem via different techniques and algorithms. we started by boosting traditional machine learning algorithms then we moved on to deep learning algorithms by proposing a bidirectional GRU (BI-GRU) model combined with word2vec, finally we also proposed a fine-tuned model of bidirectional encoder representations from Transformers (BERT). The limitations of our model’s performance as well as the lack of an annotated dataset lead us to explore a novel approach to the cognitive classification of text. First, we propose MTBERT-Attention, a unique and explainable model based on multi-task learning (MTL), BERT, and the co-attention mechanism. MTL enhances our primary task’s generalization capacity and permits data augmentation. The usage of BERT as a stack for knowledge transfer between activities has been improved. The co-attention mechanism put particular emphasis on the significant aspects of the learning objective. We propose an explainability framework based on the attention mechanism. Lastly, we carry out in-depth tests to assess the viability and efficiency of the suggested model and the explainability Framework. Our proposed model outperforms the baseline models for loss, F1-score, and accuracy. It attained an overall classification accuracy of 97.71% with the test set and effectively classifies learning objectives that use unclear action verbs from Bloom’s taxonomy. On the other hand, to prove the performance of our explainability Framework, we conduct a qualitative and quantitative study on the quality of the explanations as well as on the computational cost. We adopt Local Interpretable Model-agnostic Explanations (LIME) as a baseline for comparison. Our experiments show that our proposed approach for explainability outperforms the LIME explainer in terms of fidelity and computational cost.

Exploring interdisciplinarity of science projects based on the text mining

Interdisciplinary research has gradually become one of the main driving forces to promote original innovation of scientific research, and how to measure the interdisciplinarity of science project is becoming an important topic in the science foundation managements. Existing researches mainly using methods, such as academic degree or institutional discipline or discipline category mapping of journals, to measure the interdisciplinarity. This study proposes an approach to mine and capture the different or complementary characteristics of interdisciplinarity of projects by combining text mining and machine learning methods. First, we construct the classification system and extract a raw paper and its discipline matrix according to the discipline category of journals where the references were published in. Second, we cut the matrix to summarise the distribution of key disciplines in each paper and extract the text features in the abstract and title to form a training set. Finally, we compare and analyse the classification effects of Naive Bayesian Model, Support Vector Machine and Bidirectional Encoder Representations from Transformers (BERT) model. Then, the model evaluation indicators show that the best classification effect was achieved by the BERT model. Therefore, the deep pre-trained linguistic model BERT is chosen to predict the discipline distribution of each project. In addition, the different aspects of interdisciplinarity are measured using network coherence and discipline diversity indicators. Besides, experts are invited to evaluate and interpret the results. This proposed approach could be applied to deeply understand the discipline integration from a new perspective.