Pre-trained BERT Research Articles

Knowledge Graph Completion for High-Speed Railway Turnout Switch Machine Maintenance Based on the Multi-Level KBGC Model

The incompleteness of the existing knowledge graphs in the railway domain creates information gaps, impacting their quality and effectiveness in the operation and maintenance of high-speed railway turnout switch machines. To address this, we propose a multi-layer model (KBGC) that combines KG-BERT, graph attention network (GAT), and Convolutional Embedding Network (ConvE) for knowledge graph completion in railway maintenance. KG-BERT fine-tunes a pre-trained BERT model to extract deep semantic features from entities and relationships, converting them into graph structures. GAT captures key structural relationships between nodes using an attention mechanism, producing enriched semantic and structural embeddings. Finally, ConvE reshapes and convolves these embeddings to learn complex entity interactions, enabling accurate link prediction. Extensive experiments on the HRTOM dataset, containing triplet data from high-speed railway turnout switch machines, demonstrate the model’s effectiveness, achieving an MRR of 50.8% and a Hits@10 of 60.7%. These findings show that the KBGC model significantly improves knowledge graph completion, aiding railway maintenance personnel in decision making and preventive maintenance, and providing new tools for railway maintenance applications.

A Novel Fusion of Machine Learning Methods for Enhancing Named Entity Recognition in Indonesian Language Text

One of the important implementations in machine learning is Named Entity Recognition (NER), which is used to process text and extract entities such as people, organizations, laws, religions, and locations. NER for the Indonesian language still faces significant challenges due to the lack of high-quality labelled datasets, which limits the development of more advanced models. To address this issue, we utilized several pre-trained BERT models (bert-base-uncased, indobenchmark/indobert-base-p1, indolem/indobert-base-uncased) and datasets (NERGRIT-IndoNLU, NERGRIT-Corpus, NERUGM, and NERUI). This study proposes a novel fusion approach by integrating deep learning architectures such as CNN, Bi-LSTM, Bi-GRU, and CRF to detect 19 entities. This approach enhances BERT’s sequence modelling and feature extraction capabilities, while CRF improves entity prediction by enforcing global word-sequence constraints. Experimental results demonstrate that the fusion approach outperforms previous methods. On the bert-base-uncased dataset, accuracy reached 94.75%, while indobenchmark/indobert-base-p1 achieved 95.75%, and indolem/indobert-base-uncased achieved 95.85%. This study emphasizes the effectiveness of combining deep learning architectures with pre-trained transformers to improve NER performance in the Indonesian language. The proposed methodology offers significant advancements in entity extraction for languages with limited datasets, such as Indonesian.

Preservation of emotional context in tweet embeddings on social networking sites

In communication, emotional information is crucial, yet its preservation in tweet embeddings remains a challenge. This study aims to address this gap by exploring three distinct methods for generating embedding vectors of tweets: word2vec models, pre-trained BERT models, and fine-tuned BERT models. We conducted an analysis to assess the degree to which emotional information is conserved in the resulting embedding vectors. Our findings indicate that the fine-tuned BERT model exhibits a higher level of preservation of emotional information compared to other methods. These results underscore the importance of utilizing advanced natural language processing techniques for preserving emotional context in text data, with potential implications for enhancing sentiment analysis and understanding human communication in social media contexts.

Emotions, behaviors and places: Mapping sentiments with behaviors in Japanese tweets

Emotions are the basis of human behavior and an important part of spatial constructions. Despite numerous efforts to leverage spatio-temporal big data from social media to capture public sentiments, there is a lack of understanding of how individuals' behaviors play a role in people's sentiments at places they visit. Thus, this study aims to advance our understanding of the complex relationships between sentiments, behaviors, and places. Through a case study in Japan in 2021, we analyzed one year of countrywide geotagged social media by fine-tuning a pre-trained Japanese BERT model (accuracy: 74 %). The results emphasize the importance of understanding user emotions by considering people's behaviors, even within similar types of locations. Our research framework would be beneficial for urban planners and policymakers to enhance urban spaces.

Named Entity Recognition Using BERT

Abstract: In our current era where technology keeps increasing, where data driven applications are reign supreme, Mamed Entity Recognition (NER) emerges as an important tool, aiding in better understanding of text and powering various natural language processing tasks downstream. This paper takes into the use of pre trained BERT model for NER tasks, Skipping the need for further adjustments. Our aim is to boost the accuracy and speed of NER systems by leveraging BERT’s pre trained representations. Our approach involves leveraging built in capabilities to spot key entity labels like names, dates, and people, streaming the NER process while achieving competitive results. This research takes a step forward in maximizing the potential of pre trained models for NER applications ultimately making text analysis easier and more efficient in today’s digital world.

Enhancing Chinese–Braille translation: A two-part approach with token prediction and segmentation labeling

Visually assistive systems for the visually impaired play a pivotal role in enhancing the quality of life for the visually impaired. Assistive technologies for the visually impaired have undergone a remarkable transformation with the advent of deep learning and sophisticated assistive devices. In particular, the paper utilizes the latest machine translation models and techniques to accomplish the Chinese–Braille translation task, providing convenience for visually impaired individuals. The Traditional end-to-end Chinese–Braille translation approach incorporates Braille dots and Braille word segmentation symbols as tokens within the model’s vocabulary. However, our findings reveal that Braille word segmentation is significantly more complex than Braille dot prediction. The paper proposes a novel Two-Part Loss (TPL) method that treats these tasks distinctly, leading to significant accuracy improvements. To enhance translation performance further, we introduce a BERT-Enhanced Segmentation Transformer (BEST) method. BEST leverages knowledge distillation techniques to transfer knowledge from a pre-trained BERT model to the translate model, mitigating its limitations in word segmentation. Additionally, soft label distillation is employed to improve overall efficacy further. The TPL approach achieves an average BLEU score improvement of 1.16 and 5.42 for Transformer and GPT models on four datasets, respectively. In addition, The work presents a two-stage deep learning-based translation approach that outperforms traditional multi-step and end-to-end methods. The proposed two-stage translation method achieves an average BLEU score improvement of 0.85 across four datasets.

Analyzing public sentiment on sustainability: A comprehensive review and application of sentiment analysis techniques

In the contemporary context of escalating environmental concerns, understanding public sentiment toward sustainability initiatives is crucial for shaping effective policies and practices. This research explores the efficacy of sentiment analysis in mining social media data to gauge public attitudes toward sustainability efforts. This study employs a variety of machine learning and deep learning models to perform sentiment analysis utilizing a dataset comprising tweets related to human perception towards environmental sustainability. The aim is to transform unstructured social media text into structured sentiment scores. The comparative analysis includes pre-trained sentiment analysis models like VADER, TextBlob, and Flair with traditional machine learning models such as Logistic Regression, SVM, Decision Tree, Naive Bayes, Random Forest, alongside advanced deep learning techniques like LSTM and pre-trained models BERT and GPT-2. Our results reveal significant variations in model performance, underscoring the importance of selecting appropriate sentiment analysis tools that align with the nuanced domain of sustainability. The study further emphasizes the role of transparent and reproducible research practices in advancing trustworthy AI applications. By providing insights into public opinions on sustainability, this research contributes to the broader discourse on leveraging AI to foster environmental responsibility and action. This work not only illustrates the potential of sentiment analysis in understanding public discourse but also highlights the critical need for tailored approaches that consider the specificity of the sustainability context.

Mapping vaccine names in clinical trials to vaccine ontology using cascaded fine-tuned domain-specific language models

BackgroundVaccines have revolutionized public health by providing protection against infectious diseases. They stimulate the immune system and generate memory cells to defend against targeted diseases. Clinical trials evaluate vaccine performance, including dosage, administration routes, and potential side effects. ClinicalTrials.gov is a valuable repository of clinical trial information, but the vaccine data in them lacks standardization, leading to challenges in automatic concept mapping, vaccine-related knowledge development, evidence-based decision-making, and vaccine surveillance.ResultsIn this study, we developed a cascaded framework that capitalized on multiple domain knowledge sources, including clinical trials, the Unified Medical Language System (UMLS), and the Vaccine Ontology (VO), to enhance the performance of domain-specific language models for automated mapping of VO from clinical trials. The Vaccine Ontology (VO) is a community-based ontology that was developed to promote vaccine data standardization, integration, and computer-assisted reasoning. Our methodology involved extracting and annotating data from various sources. We then performed pre-training on the PubMedBERT model, leading to the development of CTPubMedBERT. Subsequently, we enhanced CTPubMedBERT by incorporating SAPBERT, which was pretrained using the UMLS, resulting in CTPubMedBERT + SAPBERT. Further refinement was accomplished through fine-tuning using the Vaccine Ontology corpus and vaccine data from clinical trials, yielding the CTPubMedBERT + SAPBERT + VO model. Finally, we utilized a collection of pre-trained models, along with the weighted rule-based ensemble approach, to normalize the vaccine corpus and improve the accuracy of the process. The ranking process in concept normalization involves prioritizing and ordering potential concepts to identify the most suitable match for a given context. We conducted a ranking of the Top 10 concepts, and our experimental results demonstrate that our proposed cascaded framework consistently outperformed existing effective baselines on vaccine mapping, achieving 71.8% on top 1 candidate’s accuracy and 90.0% on top 10 candidate’s accuracy.ConclusionThis study provides a detailed insight into a cascaded framework of fine-tuned domain-specific language models improving mapping of VO from clinical trials. By effectively leveraging domain-specific information and applying weighted rule-based ensembles of different pre-trained BERT models, our framework can significantly enhance the mapping of VO from clinical trials.

An aspect sentiment analysis model with Aspect Gated Convolution and Dual-Feature Filtering layers

Aspect level sentiment analysis is a basic task to determine the sentiment bias based on the contextual information near the aspect words. Some sentences contain many confusing feature words due to incomplete structure or high complexity in sentiment prediction, which can easily lead to the problem that the model pays too much attention to unimportant features. Furthermore, the role of aspect-related sentiment features is not significant in the feature extraction process. The capsule network, due to its complex network structure, leads to capsule detachment when too much information is found in the dataset. Too much computational resources are easily consumed during dynamic routing, which in turn affects the model's judgement of the polarity of aspect-related sentiment. To address these issues, we propose a capsule network (AGCDFF-Caps) with aspect-gated convolution and Dual-Feature Filtering layers for aspect-level sentiment analysis. A pre-trained BERT model is used to generate a serialised representation of the text, and the semantic representation in the contextual text is enhanced by the self-attention mechanism and bi-GRU. An aspect-gating based convolutional neural network is constructed to selectively extract contextual sentiment information about aspects and discard irrelevant information. A capsule network is then used to learn the multispatial semantic features of the aspect and the text. In particular, we incorporate a Dual-Feature Filtering network structure into the capsule network structure to strengthen the interaction between the particular aspect and the context from global and local perspectives, filtering the redundant information in local semantics and global semantics. The opinion feature representations that can more accurately express the emotional tendency of the aspect are obtained. Experimental results on SemEval2014 and Twitter datasets show that the proposed hybrid network structure has superior classification performance compared to 12 advanced baseline methods.

Impact of COVID-19 Vaccine Persuasion Strategies on Social Endorsement and Public Response on Chinese Social Media

ABSTRACT Persuading the public to get vaccinated against infectious diseases is critical and carries profound implications for preparing for future pandemics. This study examined whether and how persuasion strategies employed in pro-vaccine messages affect social endorsement and audience stance toward the COVID-19 vaccine expressed in comments on Chinese social media. Through manual coding and pre-trained BERT model, we analyzed 1,500 Weibo posts focused on COVID-19 vaccination persuasion and 238,201 associated comments. Results showed that medical experts succeeded in eliciting heightened social endorsement and receiving more pro-vaccine comments. Posts that employed negative emotional appeal were less likely to be liked or receive pro-vaccine comments. Besides, vaccine persuasion messages presented in a narrative format or emphasizing vaccine efficacy garnered significantly more likes but did not significantly receive more pro-vaccine comments. Discussing domestic issues or employing joy appeal received more pro-vaccine comments. These results offer valuable insights for health practitioners and communicators, highlighting more effective persuasion strategies for engaging citizens in vaccine-related discussions on social media. This study underscores the importance of leveraging persuasion tactics on social media to foster vaccination uptake and better prepare us for handling future pandemics.

Modified R-BERT with global semantic information for relation classification task

The objective of the relation classification task is to extract relations between entities. Recent studies have found that R-BERT (Wu and He, 2019) based on pre-trained BERT (Devlin et al., 2019) acquires extremely good results in the relation classification task. However, this method does not take into account the semantic differences between different kinds of entities and global semantic information either. In this paper, we set two different fully connected layers to take into account the semantic difference between subject and object entities. Besides, we build a new module named Concat Module to fully fuse the semantic information among the subject entity vector, object entity vector, and the whole sample sentence representation vector. In addition, we apply the average pooling to acquire a better entity representation of each entity and add the activation operation with a new fully connected layer after our Concat Module. Modifying R-BERT, we propose a new model named BERT with Global Semantic Information (GSR-BERT) for relation classification tasks. We use our approach on two datasets: the SemEval-2010 Task 8 dataset and the Chinese character relationship classification dataset. Our approach achieves a significant improvement over the two datasets. It means that our approach enjoys transferability across different datasets. Furthermore, we prove that these policies we used in our approach also enjoy applicability to named entity recognition task.

MizBERT: A Mizo BERT Model

This research investigates the utilization of pre-trained BERT transformers within the context of the Mizo language. BERT, an abbreviation for Bidirectional Encoder Representations from Transformers, symbolizes Google’s forefront neural network approach to Natural Language Processing (NLP), renowned for its remarkable performance across various NLP tasks. However, its efficacy in handling low-resource languages such as Mizo remains largely unexplored. In this study, we introduce MizBERT , a specialized Mizo language model. Through extensive pre-training on a corpus collected from diverse online platforms, MizBERT has been tailored to accommodate the nuances of the Mizo language. Evaluation of MizBERT’s capabilities is conducted using two primary metrics: masked language modeling and perplexity, yielding scores of 76.12% and 3.2565, respectively. Additionally, its performance in a text classification task is examined. Results indicate that MizBERT outperforms both the Multilingual BERT model and the Support Vector Machine algorithm, achieving an accuracy of 98.92%. This underscores MizBERT’s proficiency in understanding and processing the intricacies inherent in the Mizo language.

A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAs

Specifying legal requirements for software systems to ensure their compliance with the applicable regulations is a major concern of requirements engineering. Personal data which is collected by an organization is often shared with other organizations to perform certain processing activities. In such cases, the General Data Protection Regulation (GDPR) requires issuing a data processing agreement (DPA) which regulates the processing and further ensures that personal data remains protected. Violating GDPR can lead to huge fines reaching to billions of Euros. Software systems involving personal data processing must adhere to the legal obligations stipulated both at a general level in GDPR as well as the obligations outlined in DPAs highlighting specific business. In other words, a DPA is yet another source from which requirements engineers can elicit legal requirements. However, the DPA must be complete according to GDPR to ensure that the elicited requirements cover the complete set of obligations. Therefore, checking the completeness of DPAs is a prerequisite step towards developing a compliant system. Analyzing DPAs with respect to GDPR entirely manually is time consuming and requires adequate legal expertise. In this paper, we propose an automation strategy that addresses the completeness checking of DPAs against GDPR provisions as a text classification problem. Specifically, we pursue ten alternative solutions which are enabled by different technologies, namely traditional machine learning, deep learning, language modeling, and few-shot learning. The goal of our work is to empirically examine how these different technologies fare in the legal domain. We computed F2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$_2$$\\end{document} score on a set of 30 real DPAs. Our evaluation shows that best-performing solutions yield F2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$_2$$\\end{document} score of 86.7% and 89.7% are based on pre-trained BERT and RoBERTa language models. Our analysis further shows that other alternative solutions based on deep learning (e.g., BiLSTM) and few-shot learning (e.g., SetFit) can achieve comparable accuracy, yet are more efficient to develop.

Developing deep learning-based large-scale organic reaction classification model via sigma-profiles

Advanced technologies like deep learning have accelerated the discovery of novel chemical reactions, especially in the field of organic synthesis. With hundreds of thousands of reactions available for reference, one way to effectively leverage them is by classifying chemical reactions into different clusters based on their specific characteristics, which makes target-guided navigation in the vast chemical space possible. Although previous attempts that apply deep learning to reaction classification tasks have made substantial progress, developing a model with good interpretability as well as high accuracy for large-scale reaction classification tasks remains an open question. In this work, a deep learning-based model for a large-scale reaction classification task is first constructed by utilizing pre-trained BERT and autoencoder. Then, the model is trained under the open-source dataset USPTO_TPL which contains recorded reactions of up to 1000 different types. The multi-classification accuracy of the model on the testing dataset is 99.382%, showing its great potential for practical use. Besides, a reaction similarity map is presented to correlate the reactions in the USPTO_TPL dataset based on their sigma-profile-based statistical features. Finally, representative reactions from the testing dataset are provided to illustrate the model's effectiveness on the reaction classification task.

GraphPyRec: A novel graph-based approach for fine-grained Python code recommendation

Artificial intelligence has been widely applied in software engineering areas such as code recommendation. Significant progress has been made in code recommendation for static languages in recent years, but it remains challenging for dynamic languages like Python as accurately determining data flows before runtime is difficult. This limitation hinders data flow analysis, affecting the performance of code recommendation methods that rely on code analysis. In this study, a graph-based Python recommendation approach (GraphPyRec) is proposed by converting source code into a graph representation that captures both semantic and dynamic information. Nodes represent semantic information, with unique rules defined for various code statements. Edges depict control flow and data flow, utilizing a child-sibling-like process and a dedicated algorithm for data transfer extraction. Alongside the graph, a bag of words is created to include essential names, and a pre-trained BERT model transforms it into vectors. These vectors are integrated into a Gated Graph Neural Network (GGNN) process of the code recommendation model, enhancing its effectiveness and accuracy. To validate the proposed method, we crawled over a million lines of code from GitHub. Experimental results show that GraphPyRec outperforms existing mainstream Python code recommendation methods, achieving Top-1, 5, and 10 accuracy rates of 68.52%, 88.92%, and 94.05%, respectively, along with a Mean Reciprocal Rank (MRR) of 0.772.

Enhancing Contextual Masking in Reversible Linguistic Steganography with Ensemble Methods

Various cybercrimes can be prevented by text authentication that is responsible for preserving digital identities and contents. Digital signatures come in handy as a way of authenticating texts, which is an extensively used method. One approach to this problem is linguistic steganography, which allows hiding the signature in other words within the text and thereby facilitating efficient data management. However, it should be noted that there is a danger that these kinds of changes may result in inappropriate decisions being taken by automated computing systems not to mention change their final outputs (unseen). As such, many people are becoming more concerned with the possibility of reversing steganography so that it becomes possible to eliminate any distortions made during the process. This paper uses Contextual masking instead of masking randomly with BERT model. The goal behind this research was developing a natural language text specific Reversible Steganographic System. Our model uses pre-trained BERT as a transformer based masked language model and reversibly embeds messages through predictive word substitution. To quantify predictive uncertainty, we introduce an adaptive steganographic technique using Bayesian deep learning. This experiment shows us how our proposed system balances imperceptibility with capacity while maintaining near semantics at all times. Also, we integrate ensemble methods instead of Monte Carlo to balance the imperceptibility.

ShallowBKGC: a BERT-enhanced shallow neural network model for knowledge graph completion

Knowledge graph completion aims to predict missing relations between entities in a knowledge graph. One of the effective ways for knowledge graph completion is knowledge graph embedding. However, existing embedding methods usually focus on developing deeper and more complex neural networks, or leveraging additional information, which inevitably increases computational complexity and is unfriendly to real-time applications. In this article, we propose an effective BERT-enhanced shallow neural network model for knowledge graph completion named ShallowBKGC. Specifically, given an entity pair, we first apply the pre-trained language model BERT to extract text features of head and tail entities. At the same time, we use the embedding layer to extract structure features of head and tail entities. Then the text and structure features are integrated into one entity-pair representation via average operation followed by a non-linear transformation. Finally, based on the entity-pair representation, we calculate probability of each relation through multi-label modeling to predict relations for the given entity pair. Experimental results on three benchmark datasets show that our model achieves a superior performance in comparison with baseline methods. The source code of this article can be obtained from https://github.com/Joni-gogogo/ShallowBKGC.

DG Embeddings: The unsupervised definition embeddings learned from dictionary and glossary to gloss context words of Cloze task

For both humans and machines to acquire vocabulary, it is effective to learn words from context while using dictionaries as an auxiliary tool. It has been shown in previous linguistic studies that for humans, glossing either target words to be learned or words comprising context is an effective approach. For machines, however, previous NLP studies are mainly focused on the former. In this paper, we investigate the potentiality of context words-glossed setting. During pre-training BERT, to infuse context words with semantic features of glosses, we propose DG embeddings — the unsupervised definition embeddings learned from dictionaries and glossaries. To employ unsupervised learning is inspired by a real-world scenario of dictionary use called headword search. This can also prevent a technical duplicate from happening, as learning words from context is already based on auto-encoding models with self-supervised learning. BERT-base is used for evaluation, and we refer to BERT-base with DG embeddings as DG-BERT. According to our experimental results, compared to the vanilla BERT, DG-BERT shows the following strengths: faster pre-training convergence, noticeable improvements on various downstream tasks, a better grasp of figurative semantics, more accurate self-attention for collocation of phrases, and higher sensitivity to context words for target-word predictions in psycholinguistic diagnostics.

Machine Learning and Deep Learning Sentiment Analysis Models: Case Study on the SENT-COVID Corpus of Tweets in Mexican Spanish

This article presents a comprehensive evaluation of traditional machine learning and deep learning models in analyzing sentiment trends within the SENT-COVID Twitter corpus, curated during the COVID-19 pandemic. The corpus, filtered by COVID-19 related keywords and manually annotated for polarity, is a pivotal resource for conducting sentiment analysis experiments. Our study investigates various approaches, including classic vector-based systems such as word2vec, doc2vec, and diverse phrase modeling techniques, alongside Spanish pre-trained BERT models. We assess the performance of readily available sentiment analysis libraries for Python users, including TextBlob, VADER, and Pysentimiento. Additionally, we implement and evaluate traditional classification algorithms such as Logistic Regression, Naive Bayes, Support Vector Machines, and simple neural networks like Multilayer Perceptron. Throughout the research, we explore different dimensionality reduction techniques. This methodology enables a precise comparison among classification methods, with BETO-uncased achieving the highest accuracy of 0.73 on the test set. Our findings underscore the efficacy and applicability of traditional machine learning and deep learning models in analyzing sentiment trends within the context of low-resource Spanish language scenarios and emerging topics like COVID-19.

A topic relevance-aware click model for web search

To better understand users’ behavior patterns in web search, numerous click models are proposed to extract the implicit interaction feedback. Most existing click models are heavily based on the implicit information to model user behaviors, ignoring the impact of explicit information between queries and documents in search sessions. In this paper, we fully consider the topic relevance between queries and documents in search sessions and propose a novel topic relevance-aware click model (TRA-CM) for web search. TRA-CM consists of a relevance estimator and an examination predictor. The relevance estimator consists of a topic relevance predictor and a click context encoder. In the topic relevance predictor, we utilize the pre-trained BERT model to model the content information of queries and documents in search sessions. Meanwhile, we use transformer to encode users’ click behaviors in the click context encoder. We further apply a two-stage fusion strategy to obtain the final relevance scores. The examination predictor estimates the examination probability of each document. We further utilize learnable filters to attenuate log noise and obtain purer input features in both relevance estimator and examination predictor, and investigate different combination functions to integrate relevance scores and examination probabilities into click prediction. Extensive experiment results on two real-world session datasets prove that TRA-CM outperforms existing click models in both click prediction and relevance estimation tasks.