Bidirectional Encoder Representations From Transformers Research Articles

You are not alone: Characterizing users' relationship‐layer identities in online health communities

AbstractOnline health communities (OHCs) function as significant platforms that people use to obtain information and emotional support. Despite many studies on user behavior and relationships, little attention has been paid to user identities and how different layers of identities are interwoven. To address this potential research gap, this study examined users' relationship‐layer identities and their evolution by elaborating on the communication theory of identity (CTI) and social support theory. Additionally, based on our previous study on users' personal‐layer identities in OHCs, we investigated how users' relationship‐layer identities interacted with their personal‐layer identities. This study classified users' posts and replies into providing informational support, seeking informational support, providing emotional support, seeking emotional support, and companionship using the bidirectional encoder representation from transformers (BERT), with F1‐scores above 0.848. Through social network analysis, this study found that users of OHCs constructed their relationship‐layer identities more through informational interactions than through emotional interactions. Users with various personal‐layer identities presented different relationship‐layer identities. Users' relationships were more initiated by information exchange, and users with more interactions had more companionship activities. OHCs provided efficient communication channels for people to exchange social support.

Predicting determinants influencing user satisfaction with mental health app: An explainable machine learning approach based on unstructured data

In the contemporary digital landscape, the rising concern for mental health has sparked a surge in the use of mental health apps (MHAs) as accessible tools for addressing psychological well-being. Maintaining a high level of user satisfaction (USAT) is important for MHAs in the highly competitive app market. Leveraging BERT (Bidirectional Encoder Representations from Transformers), a state-of-the-art deep learning (DL) model, we perform topic modeling and sentiment analysis on 17,717 user online reviews. Specifically, we employ the BERTopic model to identify the determinants of USAT with MHAs. Utilizing a BERT-base-multilingual-uncase-sentiment model, we perform sentiment analysis to distinguish determinants that elicit satisfaction from those causing dissatisfaction. Also, this study tests and compares six machine learning (ML) algorithms to predict the influence of determinants on USAT with MHAs. The Light Gradient Boosting Machine (LightGBM) emerges as the top performer, showcasing its efficacy in predicting USAT determinants. By using SHAP (Shapley Additive exPlanations), an explainable ML model with cross-validation, we visualize the results of the LightGBM. The SHAP values show that the five most influential determinants of USAT with MHAs include soothing audio experience, smoking cessation support, payment and subscription management, tracking progress and mindful meditation experience. This study facilitates a deeper understanding of user experiences through the identification and prediction of determinants of USAT with MHAs. Understanding these factors and their interplay is essential for developers, clinicians, and stakeholders who aim to enhance MHAs’ services and ultimately improve the well-being of users.

Algorithmic Identification of Treatment-Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease.

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California - San Francisco (UCSF)-BERT, to identify serious AEs (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF-BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF-BERT was significantly superior at identifying hospitalization events emergent to medication use (P < 0.01). LLMs like UCSF-BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre-trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.

Public perception on active aging after COVID-19: an unsupervised machine learning analysis of 44,343 posts.

To analyze public perceptions of active aging in China on mainstream social media platforms to determine whether the "14th Five Year Plan for the Development of the Aging Career and Older Adult Care System" issued by the CPC in 2022 has fully addressed public needs. The original tweets posted on Weibo between January 1, 2020, and June 30, 2022, containing the words "aging" or "old age" were extracted. A bidirectional encoder representation from transformers (BERT)-based model was used to generate themes related to this perception. A qualitative thematic analysis and an independent review of the theme labels were conducted by the researchers. The findings indicate that public perceptions revolved around four themes: (1) health prevention and protection, (2) convenient living environments, (3) cognitive health and social integration, and (4) protecting the rights and interests of the older adult. Our study found that although the Plan aligns with most of these themes, it lacks clear planning for financial security and marital life.

Bidirectional encoders to state-of-the-art: a review of BERT and its transformative impact on natural language processing

First developed in 2018 by Google researchers, Bidirectional Encoder Representations from Transformers (BERT) represents a breakthrough in natural language processing (NLP). BERT achieved state-of-the-art results across a range of NLP tasks while using a single transformer-based neural network architecture. This work reviews BERT's technical approach, performance when published, and significant research impact since release. We provide background on BERT's foundations like transformer encoders and transfer learning from universal language models. Core technical innovations include deeply bidirectional conditioning and a masked language modeling objective during BERT's unsupervised pretraining phase. For evaluation, BERT was fine-tuned and tested on eleven NLP tasks ranging from question answering to sentiment analysis via the GLUE benchmark, achieving new state-of-the-art results. Additionally, this work analyzes BERT's immense research influence as an accessible technique surpassing specialized models. BERT catalyzed adoption of pretraining and transfer learning for NLP. Quantitatively, over 10,000 papers have extended BERT and it is integrated widely across industry applications. Future directions based on BERT scale towards billions of parameters and multilingual representations. In summary, this work reviews the method, performance, impact and future outlook for BERT as a foundational NLP technique. We provide background on BERT's foundations like transformer encoders and transfer learning from universal language models. Core technical innovations include deeply bidirectional conditioning and a masked language modeling objective during BERT's unsupervised pretraining phase. For evaluation, BERT was fine-tuned and tested on eleven NLP tasks ranging from question answering to sentiment analysis via the GLUE benchmark, achieving new state-of-the-art results. Additionally, this work analyzes BERT's immense research influence as an accessible technique surpassing specialized models. BERT catalyzed adoption of pretraining and transfer learning for NLP. Quantitatively, over 10,000 papers have extended BERT and it is integrated widely across industry applications. Future directions based on BERT scale towards billions of parameters and multilingual representations. In summary, this work reviews the method, performance, impact and future outlook for BERT as a foundational NLP technique.

BERT for Twitter Sentiment Analysis: Achieving High Accuracy and Balanced Performance

The Bidirectional Encoder Representations from Transformers (BERT) model is used in this work to analyse sentiment on Twitter data. A Kaggle dataset of manually annotated and anonymized COVID-19-related tweets was used to refine the model. Location, tweet date, original tweet content, and sentiment labels are all included in the dataset. When compared to the Multinomial Naive Bayes (MNB) baseline, BERT's performance was assessed, and it achieved an overall accuracy of 87% on the test set. The results indicated that for negative feelings, the accuracy was 0.93, the recall was 0.84, and the F1-score was 0.88; for neutral sentiments, the precision was 0.86, the recall was 0.78, and the F1-score was 0.82; and for positive sentiments, the precision was 0.82, the recall was 0.94, and the F1-score was 0.88. The model's proficiency with the linguistic nuances of Twitter, including slang and sarcasm, was demonstrated. This study also identifies the flaws of BERT and makes recommendations for future research paths, such as the integration of external knowledge and alternative designs.

Detecting cyberbullying text using the approaches with machine learning models for the low-resource Bengali language

The rising usage of social media sites and the advances in communication technologies have led to a considerable increase in cyberbullying events. Here, people are intimidated, harassed, and humiliated via digital messaging. To identify cyberbullying texts, several research have been undertaken in English and other languages with abundant resources, but relatively few studies have been conducted in low-resource languages like Bengali. This research focuses on Bengali text to find cyberbullying material by experimenting with pre-processing, feature selection, and three types of machine learning (ML) models: classical ML, deep learning (DL), and transformer learning. In classical ML, four models, support vector machine (SVM), multinomial Naive Bayes (MNB), random forest (RF), and logistic regression (LR) are used. In DL, three models, long short term memory (LSTM), Bidirectional LSTM, and convolutional neural network with bidirectional LSTM (CNN-BiLSTM) are employed. As the transformerbased pre-trained model, bidirectional encoder representations from transformers (BERT) is utilized. Using our proposed pre-processing tasks, the MNB-based approach achieves the best accuracy of 78.816% among the other classical ML models, the LSTM-based approach gains the highest result of 77.804% accuracy among the DL models, and the BERT-based approach outperforms both with 80.165% accuracy.

Probing the Representational Structure of Regular Polysemy via Sense Analogy Questions: Insights from Contextual Word Vectors.

Regular polysemes are sets of ambiguous words that all share the same relationship between their meanings, such as CHICKEN and LOBSTER both referring to an animal or its meat. To probe how a distributional semantic model, here exemplified by bidirectional encoder representations from transformers (BERT), represents regular polysemy, we analyzed whether its embeddings support answering sense analogy questions similar to "is the mapping between CHICKEN (as an animal) and CHICKEN (as a meat) similar to that which maps between LOBSTER (as an animal) to LOBSTER (as a meat)?" We did so using the LRcos model, which combines a logistic regression classifier of different categories (e.g., animal vs. meat) with a measure of cosine similarity. We found that (a) the model was sensitive to the shared structure within a given regular relationship; (b) the shared structure varies across different regular relationships (e.g., animal/meat vs. location/organization), potentially reflective of a "regularity continuum;" (c) some high-order latent structure is shared across different regular relationships, suggestive of a similar latent structure across different types of relationships; and (d) there is a lack of evidence for the aforementioned effects being explained by meaning overlap. Lastly, we found that both components of the LRcos model made important contributions to accurate responding and that a variation of this method could yield an accuracy boost of 10% in answering sense analogy questions. These findings enrich previous theoretical work on regular polysemy with a computationally explicit theory and methods, and provide evidence for an important organizational principle for the mental lexicon and the broader conceptual knowledgesystem.

Artificial intelligence: reflecting on the past and looking towards the next paradigm shift

ABSTRACT Artificial intelligence (AI) has undergone major advances over the past decades, propelled by key innovations in machine learning and the availability of big data and computing power. This paper surveys the historical progress of AI from its origins in logic-based systems like the Logic Theorist to recent deep learning breakthroughs like Bidirectional Encoder Representations from Transformers (BERT), Generative Pretrained Transformer 3 (GPT-3) and Large Language Model Meta AI (LLaMA). The early rule-based systems using handcrafted expertise gave way to statistical learning techniques and neural networks trained on large datasets. Milestones like AlexNet and AlphaGo established deep learning as a dominant AI approach. Transfer learning enabled models pre-trained on diverse corpora to excel at specialised downstream tasks. The scope of AI expanded from niche applications like playing chess to multifaceted capabilities in computer vision, natural language processing and dialogue agents. However, current AI still needs to catch up to human intelligence in aspects like reasoning, creativity, and empathy. Addressing limitations around real-world knowledge, biases, and transparency remains vital for further progress and aligning AI with human values. This survey provides a comprehensive overview of the evolution of AI and documents innovations that shaped its advancement over the past six decades.

Using the divergent association task to measure divergent thinking in Chinese elementary school students

The Divergent Association Task (DAT), published in July 2021, is a psychological test designed to measure an individual's divergent thinking. The test requires participants to name ten nouns that exhibit maximum dissimilarity from each other. The semantic distance between these nouns is then calculated to indicate the person's level of divergent thinking. In this study, we explored the applicability of the DAT for elementary school students in Chinese contexts, given that it was not initially designed for this specific population and was available only in English. We recruited a total of 348 students who were asked to complete three creativity tasks: the DAT, the Alternative Uses Task (AUT), and the Bridge-the-Associative-Gap Task (BAG). We examined the associations between DAT and the scores of the AUT and BAG tests. Moreover, we tested the accuracy of the DAT using varying numbers of nouns and different natural language processing models to calculate the semantic distance between nouns. Our findings supported the suitability of using the DAT to measure divergent thinking in elementary school students within Chinese contexts. We also found that using only eight nouns, instead of ten, could achieve a relatively high accuracy in measuring divergent thinking based on the DAT method. The language model of Word2Vec performed better than the BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) models when calculating semantic distances between nouns. This study has methodological and practical implications.

Exploring COVID-related relationship extraction: Contrasting data sources and analyzing misinformation

The COVID-19 pandemic presented an unparalleled challenge to global healthcare systems. A central issue revolves around the urgent need to swiftly amass critical biological and medical knowledge concerning the disease, its treatment, and containment. Remarkably, text data remains an underutilized resource in this context. In this paper, we delve into the extraction of COVID-related relations using transformer-based language models, including Bidirectional Encoder Representations from Transformers (BERT) and DistilBERT. Our analysis scrutinizes the performance of five language models, comparing information from both PubMed and Reddit, and assessing their ability to make novel predictions, including the detection of “misinformation.” Key findings reveal that, despite inherent differences, both PubMed and Reddit data contain remarkably similar information, suggesting that Reddit can serve as a valuable resource for rapidly acquiring information during times of crisis. Furthermore, our results demonstrate that language models can unveil previously unseen entities and relations, a crucial aspect in identifying instances of misinformation.

Research on automatic generation of multimodal medical image reports based on memory driven

The task of automatic generation of medical image reports faces various challenges, such as diverse types of diseases and a lack of professionalism and fluency in report descriptions. To address these issues, this paper proposes a multimodal medical imaging report based on memory drive method (mMIRmd). Firstly, a hierarchical vision transformer using shifted windows (Swin-Transformer) is utilized to extract multi-perspective visual features of patient medical images, and semantic features of textual medical history information are extracted using bidirectional encoder representations from transformers (BERT). Subsequently, the visual and semantic features are integrated to enhance the model's ability to recognize different disease types. Furthermore, a medical text pre-trained word vector dictionary is employed to encode labels of visual features, thereby enhancing the professionalism of the generated reports. Finally, a memory driven module is introduced in the decoder, addressing long-distance dependencies in medical image data. This study is validated on the chest X-ray dataset collected at Indiana University (IU X-Ray) and the medical information mart for intensive care chest x-ray (MIMIC-CXR) released by the Massachusetts Institute of Technology and Massachusetts General Hospital. Experimental results indicate that the proposed method can better focus on the affected areas, improve the accuracy and fluency of report generation, and assist radiologists in quickly completing medical image report writing.

Comparative study of sequence-to-sequence models: From RNNs to transformers

In this comprehensive exploration of sequence-to-sequence models in Natural Language Processing (NLP), we have traced the trajectory of their evolution and contributions. Starting from foundational Recurrent Neural Networks (RNNs) to the revolutionary capabilities of Long Short-Term Memory (LSTM), In this comprehensive exploration of sequence-to-sequence models in Natural Language Processing (NLP), we have meticulously traced the trajectory of their evolution and impactful contributions. From the foundational Recurrent Neural Networks (RNNs) to the revolutionary capabilities of Long Short-Term Memory (LSTM), as well as the transformative innovations brought forth by Transformers and BERT, this review eloquently highlights the monumental advancements that have fundamentally reshaped our understanding and generation of language. The crux of our comparative analysis lies in its ability to spotlight the distinctive strengths and limitations inherent in each model. Through an intricate examination, we uncover their nuanced applications across a diverse spectrum of NLP tasks. Particularly noteworthy is the pivotal role played by Transformers and the transformative Bidirectional Encoder Representations from Transformers (BERT). The paper concludes with a summary and outlook of the entire paper.

Spam classification based on parallel optimized BERT

With the popularity of email and the increase in spam, effectively filtering spam has become an urgent need. This research presents a cutting-edge approach to spam filtering, leveraging the power of BERT and innovative parallel optimization techniques, improving email security. This study proposes a spam classification method based on the BERT (Bidirectional Encoder Representations from Transformers) model, aiming to improve the accuracy and efficiency of spam filtering. The study first investigated the shortcomings of traditional classification methods and then analysed the applicability of the BERT model to classifying spam. The study also performed two parallel optimizations on BERT, DDP (Distributed Data Parallel), and Gpipe. Among them, DDP was used to accelerate the large model BERT parallelly. After conducting experiments on spam classification, it was found that the average training time of an epoch was 54 seconds, with the accuracy on the test set reaching 98%, achieving significant performance improvements.

An empirical study on how emotion affects the probability of replies based BERT

The rapid development of the internet, social media, and online forums have become crucial platforms for people to express their views and emotions. Comments are not only a way for users to express their opinions but also play a vital role in promoting discussions and interactions between users, significantly influencing public opinion. This paper aims to explore the impact of emotions on the likelihood of comments receiving replies, deepening the understanding of the role of emotional factors in interactions on social media and online forums. Through large-scale model training and pipeline parallel computation, this paper employs the Bidirectional Encoder Representations from Transformers (BERT) model for learning and prediction, enhancing accuracy and efficiency. The experimental results show that the response rate of negative emotional comments is about 27%, while the response rate of positive emotional comments is about 18%. It means that the comments with negative emotions are more likely to receive replies than those with positive emotions.

The Power of Context: A Novel Hybrid Context-Aware Fake News Detection Approach

The detection of fake news has emerged as a crucial area of research due to its potential impact on society. In this study, we propose a robust methodology for identifying fake news by leveraging diverse aspects of language representation and incorporating auxiliary information. Our approach is based on the utilisation of Bidirectional Encoder Representations from Transformers (BERT) to capture contextualised semantic knowledge. Additionally, we employ a multichannel Convolutional Neural Network (mCNN) integrated with stacked Bidirectional Gated Recurrent Units (sBiGRU) to jointly learn multi-aspect language representations. This enables our model to effectively identify valuable clues from news content while simultaneously incorporating content- and context-based cues, such as user posting behaviour, to enhance the detection of fake news. Through extensive experimentation on four widely used real-world datasets, our proposed framework demonstrates superior performance (↑3.59% (PolitiFact), ↑6.8% (GossipCop), ↑2.96% (FA-KES), and ↑12.51% (LIAR), considering both content-based features and additional auxiliary information) compared to existing state-of-the-art approaches, establishing its effectiveness in the challenging task of fake news detection.

Detecting multiple coexisting emotions from public emergency opinions

To detect multiple coexisting emotions from public emergency opinions, this article proposes a novel two-stage multiple coexisting emotion-detection model. First, the text semantic feature extracted through bidirectional encoder representation from transformers (BERT) and the emotion lexicon feature extracted through the emotion dictionary are fused. Then, the emotion subjectivity judgement and multiple coexisting emotion detection are performed in two separate stages. In the first stage, we introduce synthetic minority oversampling technique (SMOTE) to enhance the balance of data distribution and select the optimal classifier to recognise opinion texts with emotion. In the second stage, the label powerset (LP)-SMOTE is proposed to increase the number of the minority category samples, and multichannel emotion classifiers and the decision mechanism are employed to recognise different types of emotions and determine the final coexisting emotion labels. Finally, the Weibo data about coronavirus disease 2019 (COVID-19) are collected to verify the effectiveness of the proposed model. Experiment results indicate that the proposed model outperforms state-of-the-art models, with the F1_macro of 0.8532, the F1_micro of 0.8333, and the hamming loss of 0.0476. The emotion detection results are conducive to decision-making for public emergency departments.

Comparative Analysis of Word Embeddings for Multiclass Cyberbullying Detection

Cyberbullying has emerged as a pervasive concern in modern society, particularly within social media platforms. This phenomenon encompasses employing digital communication to instill fear, threaten, harass, or harm individuals. Given the prevalence of social media in our lives, there is an escalating need for effective methods to detect and combat cyberbullying. This paper aims to explore the utilization of word embeddings and to discern the comparative effectiveness of trainable word embeddings, pre-trained word embeddings, and fine-tuned language models in multiclass cyberbullying detection. Distinguishing from previous binary classification methods, our research delves into nuanced multiclass detection. The exploration of word embeddings holds significant promise due to its ability to transform words into dense numerical vectors within a high-dimensional space. This transformation captures intricate semantic and syntactic relationships inherent in language, enabling machine learning (ML) algorithms to discern patterns that might signify cyberbullying. In contrast to previous research, this work delves beyond primary binary classification and centers on the nuanced realm of multiclass cyberbullying detection. The research employs diverse techniques, including convolutional neural networks and bidirectional long short-term memory, alongside well-known pre-trained models such as word2vec and bidirectional encoder representations from transformers (BERT). Moreover, traditional ML algorithms such as K-nearest neighbors, Random Forest, and Naïve Bayes are integrated to evaluate their performance vis-à-vis deep learning models. The findings underscore the promise of a fine-tuned BERT model on our dataset, yielding the most promising results in multiclass cyberbullying detection, and achieving the best-recorded accuracy of 85% on the dataset.

Research on the identification and evolution of health industry policy instruments in China.

The application of health industry policies could be discovered more quickly and comprehensively through the automated identification of policy tools, which could provide references for the formulation, implementation, and optimization of subsequent policies in each province. This study applies the Bidirectional Encoder Representation from Transformer (BERT) model to identify policy tools automatically, utilizes Focal Loss to reduce the unbalance of a dataset, and analyzes the evolution of policy tools in each province, which contains time, space, and topic. The research demonstrates that the BERT model can improve the accuracy of classification, that supply and environment policy tools are more prevalent than demand tools, and that policy instruments are organized similarly in four major economic regions. Moreover, the policy's attention to topics related to healthcare, medicine, and pollution has gradually shifted to other topics, and the extent of policy attention continues to be concentrated on the health service industry, with less attention paid to the manufacturing industry from the keywords of the various topics.

Categorization of tweets for damages: infrastructure and human damage assessment using fine-tuned BERT model.

Identification of infrastructure and human damage assessment tweets is beneficial to disaster management organizations as well as victims during a disaster. Most of the prior works focused on the detection of informative/situational tweets, and infrastructure damage, only one focused on human damage. This study presents a novel approach for detecting damage assessment tweets involving infrastructure and human damages. We investigated the potential of the Bidirectional Encoder Representations from Transformer (BERT) model to learn universal contextualized representations targeting to demonstrate its effectiveness for binary and multi-class classification of disaster damage assessment tweets. The objective is to exploit a pre-trained BERT as a transfer learning mechanism after fine-tuning important hyper-parameters on the CrisisMMD dataset containing seven disasters. The effectiveness of fine-tuned BERT is compared with five benchmarks and nine comparable models by conducting exhaustive experiments. The findings show that the fine-tuned BERT outperformed all benchmarks and comparable models and achieved state-of-the-art performance by demonstrating up to 95.12% macro-f1-score, and 88% macro-f1-score for binary and multi-class classification. Specifically, the improvement in the classification of human damage is promising.