Bidirectional Encoder Representations From Transformers Research Articles

Hidden emotional trends on social media regarding the Thailand–China high-speed railway project: a deep learning approach with ChatGPT integration

Leveraging sentiment analysis on social media reveals hidden emotional trends by providing deep insights into the public’s collective opinions and feelings because social media can reflect most opinions based on true feelings. However, there is no time-series analysis of Thai opinions on social media regarding government projects in Thailand. Therefore, the focus of this research is to survey and dynamically analyze sentiment on social media platforms, particularly YouTube, regarding the Thailand–China High-Speed Rail Project (Thai–China HSR Project). This is a major project involving an investment of over 500 billion baht, a collaborative effort between the Thai and Chinese governments aimed at developing transportation infrastructure. It leads the public concerns of Thai people about the potential economic and social impacts. The deep learning (DL) technique was used for this analysis by the Bidirectional Encoder Representations from Transformers (BERT) model, together with WangchanBERTa, a Thai language model specifically designed to analyze opinions from Thai comments received from YouTube from July 2015 to January 2024. The analysis reveals that citizens’ sentiments are primarily neutral, with more negative than positive sentiments with the passing years. Results also indicate that the data volume has increased since 2019 and peaked in 2022. Negative comments peaked in January 2022, and based on our analysis, such negative comments followed news covering the opening of the China–Laos Railway on December 1, 2021. Video contents were related to the Thai–China HSR Project, which provides comparative reviews and analysis of government operations. Additionally, we explore the possibility of integrating AI technology based on the ChatGPT 4 model to support data interpretation. Although the AI model can interpret data quickly and efficiently serve as a convenient method for users, the results lack the in-depth analysis that should be done by experts in the field. The proposed framework could be useful to government agencies and can be used to quickly and efficiently survey and track public opinion. The results can be used for policy planning, strategic decision-making, and improving communications to better inform citizens about projects and operations, leading to the country’s sustainable economic and social development.

Multi-Feature Fusion in Graph Convolutional Networks for Data Network Propagation Path Tracing

With the rapid development of information technology, the complexity of data networks is increasing, especially in electric power systems, where data security and privacy protection are of great importance. Throughout the entire distribution process of the supply chain, it is crucial to closely monitor the propagation paths and dynamics of electrical data to ensure security and quickly initiate comprehensive traceability investigations if any data tampering is detected. This research addresses the challenges of data network complexity and its impact on the security of power systems by proposing an innovative data network propagation path tracing model, which is constructed based on graph convolutional networks (GCNs) and the BERT model. Firstly, propagation trees are constructed based on the propagation structure, and the key attributes of data nodes are extracted and screened. Then, GCNs are utilized to learn the representation of node features with different attribute feature combinations in the propagation path graph, while the Bidirectional Encoder Representations from Transformers (BERT) model is employed to capture the deep semantic features of the original text content. The core of this research is to effectively integrate these two feature representations, namely the structural features obtained by GCNs and the semantic features obtained by the BERT model, in order to enhance the ability of the model to recognize the data propagation path. The experimental results demonstrate that this model performs well in power data propagation and tracing tasks, and the data recognition accuracy reaches 92.5%, which is significantly better than the existing schemes. This achievement not only improves the power system’s ability to cope with data security threats but also provides strong support for protecting data transmission security and privacy.

Movie Review Based Sentiment Analysis

Sentiment analysis, an automated method for detecting emotions and opinions in text, has become a versatile tool applied across various domains, from assessing online customer feedback to monitoring sentiment trends on social media. While Bag of Words (BoW) has been the traditional method, more advanced techniques like Bidirectional Encoder Representations from Transformers (BERT) and Long Short-Term Memory (LSTM) have revolutionized sentiment analysis by elevating accuracy and context comprehension. This study conducts an extensive comparison between BERT, a hybrid model that combines CNN and LSTM in the realm of sentiment analysis within movie reviews. Through meticulous experimentation, the study strongly proves that both the hybrid model of CNN and LSTM models significantly outshine the techniques like BERT and BoW with noticeably higher accuracy in precisely categorizing sentiments expressed in movie reviews. These results highlight how deep learning methodologies remarkably refine the precision and effectiveness of sentiment analysis, allowing for more nuanced and context-aware sentiment classification. The study's significance goes beyond performance contrasts; it demonstrates the exceptional capacity of BERT along with the hybrid model of CNN and LSTM to comprehend the complexities of language and contextual nuances within movie reviews. This heightened contextual understanding is particularly crucial in sentiment analysis, enabling the models to discern subtle shifts in sentiment that simpler methods like BoW might overlook. By showcasing the supremacy of hybrid model that combines CNN and LSTM in movie review sentiment analysis, this research opens new avenues for the field's advancement. It emphasizes the potential of deep learning techniques in revolutionizing sentiment analysis across diverse domains, offering more accurate and contextually attuned sentiment classification for deeper insights and well-informed decision-making. This study stands as a valuable asset for researchers and practitioners aiming to leverage deep learning for sentiment analysis applications

Unmasking large language models by means of OpenAI GPT-4 and Google AI: A deep instruction-based analysis

Large Language Models (LLMs) have become a hot topic in AI due to their ability to mimic human conversation. This study compares the open artificial intelligence generative pretrained transformer-4 (GPT-4) model, based on the (GPT), and Google's artificial intelligence (AI), which is based on the Bidirectional Encoder Representations from Transformers (BERT) framework in terms of the defined capabilities and the built-in architecture. Both LLMs are prominent in AI applications. First, eight different capabilities were identified to evaluate these models, i.e. translation accuracy, text generation, factuality, creativity, intellect, deception avoidance, sentiment classification, and sarcasm detection. Next, each capability was assessed using instructions. Additionally, a categorized LLM evaluation system has been developed by means of using ten research questions per category based on this paper's main contributions from a prompt engineering perspective. It should be highlighted that GPT-4 and Google AI successfully answered 85 % and 68,7 % of the study prompts, respectively. It has been noted that GPT-4 better understands prompts than Google AI, even with verbal flaws, and tolerates grammatical errors. Moreover, the GPT-4 based approach was more precise, accurate, and succinct than Google AI, which was sometimes verbose and less realistic. While GPT-4 beats Google AI in terms of translation accuracy, text generation, factuality, intellectuality, creativity, and deception avoidance, Google AI outperforms the former when considering sarcasm detection. Both sentiment classification models did work properly. More importantly, a human panel of judges was used to assess and evaluate the model comparisons. Statistical analysis of the judges' ratings revealed more robust results based on examining the specific uses, limitations, and expectations of both GPT-4 and Google AI-based approaches. Finally, the two approaches' transformers, parameter sizes, and attention mechanisms have been examined.

Language conditioned multi-scale visual attention networks for visual grounding

Visual grounding (VG) is a task that requires to locate a specific region in an image according to a natural language expression. Existing efforts on the VG task are divided into two-stage, one-stage and Transformer-based methods, which have achieved high performance. However, most of the previous methods extract visual information at a single spatial scale and ignore visual information at other spatial scales, which makes these models unable to fully utilize the visual information. Moreover, the insufficient utilization of linguistic information, especially failure to capture global linguistic information, may lead to failure to fully understand language expressions, thus limiting the performance of these models. To better address the task, we propose a language conditioned multi-scale visual attention network (LMSVA) for visual grounding, which can sufficiently utilize visual and linguistic information to perform multimodal reasoning, thus improving performance of model. Specifically, we design a visual feature extractor containing a multi-scale layer to get the required multi-scale visual features by expanding the original backbone. Moreover, we exploit pooling the output of the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to extract sentence-level linguistic features, which can enable the model to capture global linguistic information. Inspired by the Transformer architecture, we present the Visual Attention Layer guided by Language and Multi-Scale Visual Features (VALMS), which is able to better learn the visual context guided by multi-scale visual and linguistic features, and facilitates further multimodal reasoning. Extensive experiments on four large benchmark datasets, including ReferItGame, RefCOCO, RefCOCO+ and RefCOCOg, demonstrate that our proposed model achieves the state-of-the-art performance.

Development of an Internet-based Product-related Child Injury Textual Data Platform (IPCITDP) in China.

Internet-based media stories provide valuable information for emerging risks of product-related child injury prevention and control, but critical methodological challenges and high costs of data acquisition and processing restrict practical use by stakeholders. We constructed a data platform through literature reviews and multi-round research group discussions. Developed components included standard search strategies, filtering criteria, textual document classification, information extraction standards and a keyword dictionary. We used ten thousand manually labelled media stories to validate the textual document classification model, which was established using the Bidirectional Encoder Representation from Transformers (BERT). Multiple information extraction methods based on natural language processing algorithms were adopted to extract data for 29 structured variables from media stories. They were evaluated through manual validation of 1000 media stories about product-related child injury. We mapped the geographic distribution of media sources and media-reported product-related child injury events. We developed an internet-based product-related child injury textual data platform, IPCITDP, consisting of four layers - automatic data search and acquisition, data processing, data storage, and data application - concerning product-related child injury online media stories in China. Each layer occurred daily. External validation demonstrated high performance for the BERT classification model we established (accuracy = 0.9703) and the combined information extraction strategies (accuracy >0.70 for 25 variables). As of 31 December 2023, IPCITDP collected 35 275 eligible product-related child injury reports from 13 261 news media websites or social media platform accounts which were geographically located across all 31 mainland Chinese provinces and covered over 97% of prefecture-level cities. The injury cases in IPCITDP were typically reported several months or years earlier than official announcements about the product-related child injury risks. Our data platform added data concerning 15 supplementary variables that the national product-related injury surveillance system lacks. Two examples demonstrate the value of IPCITDP in supplementing existing data and providing early epidemiological detection of emerging signals concerning product-related child injury: magnetic beads and electric self-balancing scooters. Our data platform provides injury data that can support early detection of new product-related child injury characteristics and supplement existing data sources to reduce the burden of product-related injury among Chinese children.

PhosBERT: A self-supervised learning model for identifying phosphorylation sites in SARS-CoV-2-infected human cells

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a single-stranded RNA virus, which mainly causes respiratory and enteric diseases and is responsible for the outbreak of coronavirus disease 19 (COVID-19). Numerous studies have demonstrated that SARS-CoV-2 infection will lead to a significant dysregulation of protein post-translational modification profile in human cells. The accurate recognition of phosphorylation sites in host cells will contribute to a deep understanding of the pathogenic mechanisms of SARS-CoV-2 and also help to screen drugs and compounds with antiviral potential. Therefore, there is a need to develop cost-effective and high-precision computational strategies for specifically identifying SARS-CoV-2-infected phosphorylation sites. In this work, we first implemented a custom neural network model (named PhosBERT) on the basis of a pre-trained protein language model of ProtBert, which was a self-supervised learning approach developed on the Bidirectional Encoder Representation from Transformers (BERT) architecture. PhosBERT was then trained and validated on serine (S) and threonine (T) phosphorylation dataset and tyrosine (Y) phosphorylation dataset with 5-fold cross-validation, respectively. Independent validation results showed that PhosBERT could identify S/T phosphorylation sites with high accuracy and AUC (area under the receiver operating characteristic) value of 81.9% and 0.896. The prediction accuracy and AUC value of Y phosphorylation sites reached up to 87.1% and 0.902. It indicated that the proposed model was of good prediction ability and stability and would provide a new approach for studying SARS-CoV-2 phosphorylation sites.

Social Network Public Opinion Analysis Using BERT-BMA in Big Data Environment

The existing social network public opinion analysis methods have problems such as poor semantic expression quality and weak detection ability in short texts. Therefore, a social network public opinion analysis method based on BERT-BMA is proposed. To normalize the comment text, the rumor text is initially transferred to a word vector matrix using the BERT (Bidirectional Encoder Representations from Transformer) model. The BiLSTM-based network architecture is subsequently employed to acquire the trace features of data transmission. Ultimately, this study employs the multi-head attention mechanism to extract feature information that is more significant in the analysis of online public opinion by mining the dependency relationships between users, resulting in increasing ability to detect public opinion emergencies. The experimental outcomes indicate that the results on the Twitter data set and Weibo dataset are superior to other comparative models.

Dynamic elicitation and forecasting innovation requirement of smart product-service system via user-manufacturer value co-creation perspective using multi-source data

Achieving greater efficiency and lower resource consumption is a constant pursuit for successfully developing smart product-service system (PSS). Conducting dynamic requirements analysis is increasingly important in the process while the value co-creation of smart PSS stakeholders has not been sufficiently considered. This study adopts a user-manufacturer value co-creation perspective to explore smart PSS dynamic requirement elicitation and forecasting. Using multi-source data of user online reviews, historical descriptive texts and patent abstracts, this study dynamically elicits the smart PSS innovation requirements using dynamic topic model (DTM) and bidirectional encoder representation from transformers (BERT). Furthermore, the key innovation requirements of smart PSS are identified by integrating their importance and satisfaction values into a dynamic importance-performance analysis (DIPA). Then, the trend of smart PSS key innovation requirements is predicted by the grey forecasting model to determine the smart PSS improvement direction. An empirical study of smartwatch service system is conducted to validate the proposed approach. By blending both user and manufacturer requirements, this study provides a novel framework for dynamic requirement elicitation and forecasting of smart PSS based on multi-source data. It contributes to facilitating the value co-creation during smart PSS development to optimize manufacturer’s R&D resource allocation and enhance user experience.

Multi-modal deep learning framework for damage detection in social media posts.

In crisis management, quickly identifying and helping affected individuals is key, especially when there is limited information about the survivors' conditions. Traditional emergency systems often face issues with reachability and handling large volumes of requests. Social media has become crucial in disaster response, providing important information and aiding in rescues when standard communication systems fail. Due to the large amount of data generated on social media during emergencies, there is a need for automated systems to process this information effectively and help improve emergency responses, potentially saving lives. Therefore, accurately understanding visual scenes and their meanings is important for identifying damage and obtaining useful information. Our research introduces a framework for detecting damage in social media posts, combining the Bidirectional Encoder Representations from Transformers (BERT) architecture with advanced convolutional processing. This framework includes a BERT-based network for analyzing text and multiple convolutional neural network blocks for processing images. The results show that this combination is very effective, outperforming existing methods in accuracy, recall, and F1 score. In the future, this method could be enhanced by including more types of information, such as human voices or background sounds, to improve its prediction efficiency.

Deciphering 3'UTR Mediated Gene Regulation Using Interpretable Deep Representation Learning.

The 3' untranslated regions (3'UTRs) of messenger RNAs contain many important cis-regulatory elements that are under functional and evolutionary constraints. It is hypothesized that these constraints are similar to grammars and syntaxes in human languages and can be modeled by advanced natural language techniques such as Transformers, which has been very effective in modeling complex protein sequence and structures. Here 3UTRBERT is described, which implements an attention-based language model, i.e., Bidirectional Encoder Representations from Transformers (BERT). 3UTRBERT is pre-trained on aggregated 3'UTR sequences of human mRNAs in a task-agnostic manner; the pre-trained model is then fine-tuned for specific downstream tasks such as identifying RBP binding sites, m6A RNA modification sites, and predicting RNA sub-cellular localizations. Benchmark results show that 3UTRBERT generally outperformed other contemporary methods in each of these tasks. More importantly, the self-attention mechanism within 3UTRBERT allows direct visualization of the semantic relationship between sequence elements and effectively identifies regions with important regulatory potential. It is expected that 3UTRBERT model can serve as the foundational tool to analyze various sequence labeling tasks within the 3'UTR fields, thus enhancing the decipherability of post-transcriptional regulatorymechanisms.

Enhancing E-Commerce Recommendation Systems with Multiple Item Purchase Data: A Bidirectional Encoder Representations from Transformers-Based Approach

This study proposes how to incorporate concurrent purchase data into e-commerce recommendation systems to improve their predictive accuracy. We identified that concurrent purchases account for about 23% of total orders on Katcher’s, a Korean e-commerce platform. Despite the prevalence of concurrent purchases, existing algorithms often overlook this aspect. We introduce a novel transformer-based recommendation algorithm to process a user’s order history, including concurrent purchases. Each order is represented as a natural language sentence, capturing the order timestamp, product names and their attribute values, their corresponding categories, and whether multiple products were purchased together in a single order (i.e., a concurrent purchase). These sentences form a sequence, which serves as a training dataset for fine-tuning Bidirectional Encoder Representations from Transformers (BERT) with the Next Sentence Prediction objective. We validate our ideas by conducting experiments on Katcher’s platform, demonstrating the proposed method’s improved prediction performance compared to existing recommendation systems, with enhanced accuracy and F1 score. Notably, the normalized discounted cumulative gain (NDCG) showed a significant improvement with a large margin. Furthermore, we demonstrate the beneficial impact of integrating concurrent purchase information on prediction performance.

Accelerating the discovery of acceptor materials for organic solar cells by deep learning

It is a time-consuming and costly process to develop affordable and high-performance organic photovoltaic materials. Computational methods are essential for accelerating the material discovery process by predicting power conversion efficiencies (PCE). In this study, we propose a deep learning-based framework (DeepAcceptor) to design and discover highly efficient small molecule acceptor materials. Specifically, an experimental dataset is constructed by collecting acceptor data from publications. Then, a deep learning-based model is customized to predict PCEs by applying graph representation learning to Bidirectional Encoder Representations from Transformers (BERT), with the atom, bond, and connection information in acceptor molecular structures as the input (abcBERT). The computational dataset derived from density functional theory (DFT) calculations and the experimental dataset from literature are used to pre-train and fine-tune the model, respectively. The abcBERT model outperforms other state-of-the-art models for the PCE prediction with MAE = 1.78 and R2 = 0.67 on the test set. A molecular generation and screening process is built to find new high-performance acceptors for PM6. Three discovered candidates are further validated by experiment, and the best PCE reaches 14.61%. The released user-friendly interface of DeepAcceptor greatly boosts the accessibility and efficiency of designing and discovering high-performance acceptors. Altogether, the DeepAcceptor framework with abcBERT is promising to predict the PCE and accelerate the discovery of high-performance acceptor materials.

Utilizing Large Language Models to Illustrate Constraints for Construction Planning

Effective construction project planning relies on addressing constraints related to materials, labor, equipment, and others. Planning meetings are typical venues for stakeholders to identify, communicate, and remove constraints. However, a critical gap exists in lacking an automated approach to identify, classify, analyze, and track constraint discussions during onsite planning meetings. Therefore, this research aims to 1. develop a natural language processing model to classify constraints in meeting discussions; 2. uncover the discussion patterns of managers and foremen regarding various constraints; and 3. extract the root causes for constraints, evaluate their impacts, and prepare managers to develop practical solutions for constraint removal. This research collected meeting transcripts from 94 onsite planning meetings of a building project, spanning 263,836 words. Next, this research leveraged a general pretrained transformer (GPT) to segment discussion dialogs into topics. A Bidirectional Encoder Representations from Transformers (BERT)-based model was developed to categorize constraint types for each topic. The constraint patterns among meeting attendees were assessed. Furthermore, a GPT-based tool was devised to track root causes, impacts, and solutions for various constraints. Test results revealed an 8.8% improvement in constraint classification accuracy compared with the traditional classification model. An occupational characteristic in constraint discussion was observed in that the management team tended to balance their focus on various constraints, while foremen concentrated on more practical issues. This research contributes to the body of knowledge by leveraging language models to analyze construction planning meetings. The findings facilitate project managers in establishing constraint logs for diagnosing and prognosticating planning issues.

Cancer Prevention and Treatment on Chinese Social Media: Machine Learning-Based Content Analysis Study.

Nowadays, social media plays a crucial role in disseminating information about cancer prevention and treatment. A growing body of research has focused on assessing access and communication effects of cancer information on social media. However, there remains a limited understanding of the comprehensive presentation of cancer prevention and treatment methods across social media platforms. Furthermore, research comparing the differences between medical social media (MSM) and common social media (CSM) is also lacking. Using big data analytics, this study aims to comprehensively map the characteristics of cancer treatment and prevention information on MSM and CSM. This approach promises to enhance cancer coverage and assist patients in making informed treatment decisions. We collected all posts (N=60,843) from 4 medical WeChat official accounts (accounts with professional medical backgrounds, classified as MSM in this paper) and 5 health and lifestyle WeChat official accounts (accounts with nonprofessional medical backgrounds, classified as CSM in this paper). We applied latent Dirichlet allocation topic modeling to extract cancer-related posts (N=8427) and identified 6 cancer themes separately in CSM and MSM. After manually labeling posts according to our codebook, we used a neural-based method for automated labeling. Specifically, we framed our task as a multilabel task and utilized different pretrained models, such as Bidirectional Encoder Representations from Transformers (BERT) and Global Vectors for Word Representation (GloVe), to learn document-level semantic representations for labeling. We analyzed a total of 4479 articles from MSM and 3948 articles from CSM related to cancer. Among these, 35.52% (2993/8427) contained prevention information and 44.43% (3744/8427) contained treatment information. Themes in CSM were predominantly related to lifestyle, whereas MSM focused more on medical aspects. The most frequently mentioned prevention measures were early screening and testing, healthy diet, and physical exercise. MSM mentioned vaccinations for cancer prevention more frequently compared with CSM. Both types of media provided limited coverage of radiation prevention (including sun protection) and breastfeeding. The most mentioned treatment measures were surgery, chemotherapy, and radiotherapy. Compared with MSM (1137/8427, 13.49%), CSM (2993/8427, 35.52%) focused more on prevention. The information about cancer prevention and treatment on social media revealed a lack of balance. The focus was primarily limited to a few aspects, indicating a need for broader coverage of prevention measures and treatments in social media. Additionally, the study's findings underscored the potential of applying machine learning to content analysis as a promising research approach for mapping key dimensions of cancer information on social media. These findings hold methodological and practical significance for future studies and health promotion.

Multimodal Religiously Hateful Social Media Memes Classification Based on Textual and Image Data

Multimodal hateful social media meme detection is an important and challenging problem in the vision-language domain. Recent studies show high accuracy for such multimodal tasks due to datasets that provide better joint multimodal embedding to narrow the semantic gap. Religiously hateful meme detection is not extensively explored among published datasets. While there is a need for higher accuracy on religiously hateful memes, deep learning–based models often suffer from inductive bias. This issue is addressed in this work with the following contributions. First, a religiously hateful memes dataset is created and published publicly to advance hateful religious memes detection research. Over 2000 meme images are collected with their corresponding text. The proposed approach compares and fine-tunes VisualBERT pre-trained on the Conceptual Caption (CC) dataset for the downstream classification task. We also extend the dataset with the Facebook hateful memes dataset. We extract visual features using ResNeXT-152 Aggregated Residual Transformations–based Masked Regions with Convolutional Neural Networks (R-CNN) and Bidirectional Encoder Representations from Transformers (BERT) uncased for textual encoding for the early fusion model. We use the primary evaluation metric of an Area Under the Operator Characters Curve (AUROC) to measure model separability. Results show that the proposed approach has a higher AUROC score of 78%, proving the model’s higher separability performance and an accuracy of 70%. It shows comparatively superior performance considering dataset size and against ensemble-based machine learning approaches.

Natural Language Processing Pretraining Language Model for Computer Intelligent Recognition Technology

Computer intelligent recognition technology refers to the use of computer vision, Natural Language Processing (NLP), machine learning and other technologies to enable computers to recognize, analyze, understand and answer human language and behavior. The common applications of computer intelligent recognition technology include image recognition, NLP, face recognition, target tracking, and other fields. NLP is a field of computer science, which involves the interaction between computers and natural languages. NLP technology can be used to process, analyze and generate natural language data, such as text, voice and image. Common NLP technology applications include language translation, emotion analysis, text classification, speech recognition and question answering system. Language model is a machine learning model, which uses a large number of text data for training to learn language patterns and relationships in text data. Although the language model has made great progress in the past few years, it still faces some challenges, including: poor semantic understanding, confusion in multilingual processing, slow language processing and other shortcomings. Therefore, in order to optimize these shortcomings, this article would study the pre-training language model based on NLP technology, which aimed at using NLP technology to optimize and improve the performance of the language model, thus optimizing the computer intelligent recognition technology. The model had a higher language understanding ability and more accurate prediction ability. In addition, the model could learn language rules and structures by using a large number of corpus, so as to better understand natural language. Through experiments, it could be known that the data size and total computing time of the traditional Generative Pretrained Transformer-2 (GPT-2) language model were 10 GB and 97 hours respectively. The data size and total computing time of BERT (Bidirectional Encoder Representations from Transformer) were 12 GB and 86 hours respectively. The data size and total computing time of the pre-training language model based on NLP were 18 GB and 71 hours respectively. Obviously, the pre-training language model based on NLP had a larger data size and shorter computing time. The experimental results showed that the NLP technology could better optimize the language model and effectively improve its various capabilities. This article opened up a new development direction for computer intelligent recognition technology and provided excellent technical support for the development of language models.

Natural language processing with transformers: a review.

Natural language processing (NLP) tasks can be addressed with several deep learning architectures, and many different approaches have proven to be efficient. This study aims to briefly summarize the use cases for NLP tasks along with the main architectures. This research presents transformer-based solutions for NLP tasks such as Bidirectional Encoder Representations from Transformers (BERT), and Generative Pre-Training (GPT) architectures. To achieve that, we conducted a step-by-step process in the review strategy: identify the recent studies that include Transformers, apply filters to extract the most consistent studies, identify and define inclusion and exclusion criteria, assess the strategy proposed in each study, and finally discuss the methods and architectures presented in the resulting articles. These steps facilitated the systematic summarization and comparative analysis of NLP applications based on Transformer architectures. The primary focus is the current state of the NLP domain, particularly regarding its applications, language models, and data set types. The results provide insights into the challenges encountered in this research domain.

E-Commerce Live Streaming Danmaku Classification Through LDA-Enhanced BERT-TextCNN Model

With the increasing popularity of e-commerce live streaming, comprehensive analysis of live barrage text has become increasingly crucial. This study presents a thematic analysis method for categorizing e-commerce live streaming barrage text using latent Dirichlet allocation (LDA) topic modeling, combined with the advantages of the Bidirectional Encoder Representations from Transformers (BERT) and TextCNN models. The LDA algorithm is initially used to extract topics from the barrage text, and then a dataset comprising six designated categories is assembled. Subsequently, a BERT-TextCNN hybrid model is trained, merging BERT's profound semantic comprehension with TextCNN's ability to extract local features. Empirical evidence shows that the proposed model notably improves classification accuracy and efficiency, providing valuable theoretical and practical insights for crafting strategies and optimizing user experience on e-commerce live streaming platforms.

Enhancing Clinical Relevance of Pretrained Language Models Through Integration of External Knowledge: Case Study on Cardiovascular Diagnosis From Electronic Health Records.

Despite their growing use in health care, pretrained language models (PLMs) often lack clinical relevance due to insufficient domain expertise and poor interpretability. A key strategy to overcome these challenges is integrating external knowledge into PLMs, enhancing their adaptability and clinical usefulness. Current biomedical knowledge graphs like UMLS (Unified Medical Language System), SNOMED CT (Systematized Medical Nomenclature for Medicine-Clinical Terminology), and HPO (Human Phenotype Ontology), while comprehensive, fail to effectively connect general biomedical knowledge with physician insights. There is an equally important need for a model that integrates diverse knowledge in a way that is both unified and compartmentalized. This approach not only addresses the heterogeneous nature of domain knowledge but also recognizes the unique data and knowledge repositories of individual health care institutions, necessitating careful and respectful management of proprietary information. This study aimed to enhance the clinical relevance and interpretability of PLMs by integrating external knowledge in a manner that respects the diversity and proprietary nature of health care data. We hypothesize that domain knowledge, when captured and distributed as stand-alone modules, can be effectively reintegrated into PLMs to significantly improve their adaptability and utility in clinical settings. We demonstrate that through adapters, small and lightweight neural networks that enable the integration of extra information without full model fine-tuning, we can inject diverse sources of external domain knowledge into language models and improve the overall performance with an increased level of interpretability. As a practical application of this methodology, we introduce a novel task, structured as a case study, that endeavors to capture physician knowledge in assigning cardiovascular diagnoses from clinical narratives, where we extract diagnosis-comment pairs from electronic health records (EHRs) and cast the problem as text classification. The study demonstrates that integrating domain knowledge into PLMs significantly improves their performance. While improvements with ClinicalBERT are more modest, likely due to its pretraining on clinical texts, BERT (bidirectional encoder representations from transformer) equipped with knowledge adapters surprisingly matches or exceeds ClinicalBERT in several metrics. This underscores the effectiveness of knowledge adapters and highlights their potential in settings with strict data privacy constraints. This approach also increases the level of interpretability of these models in a clinical context, which enhances our ability to precisely identify and apply the most relevant domain knowledge for specific tasks, thereby optimizing the model's performance and tailoring it to meet specific clinical needs. This research provides a basis for creating health knowledge graphs infused with physician knowledge, marking a significant step forward for PLMs in health care. Notably, the model balances integrating knowledge both comprehensively and selectively, addressing the heterogeneous nature of medical knowledge and the privacy needs of health care institutions.