Pre-trained BERT Model Research Articles

Improving edit-based unsupervised sentence simplification using fine-tuned BERT

Word suggestion in unsupervised sentence simplification aims to replace complex words of a given sentence with their simpler alternatives. This is mostly done without considering their context within the input sentence. In this paper, we propose a technique that brings context awareness to word suggestion by merging pre-trained BERT models with a successful edit-based unsupervised sentence simplification model. More importantly, we show that only by fine-tuning the BERT model on simple English corpora, simplification results can be improved and even outperform some of the competing supervised methods. Finally, we introduce a framework that involves filtering an arbitrary amount of unlabeled in-domain text for tuning the model in situations where labeled data, as simple and complex, is scarce. This preprocessing step also speeds up the training process by ignoring fine-tuning on unnecessary samples.

Vulnerability Detection of Ethereum Smart Contract Based on SolBERT-BiGRU-Attention Hybrid Neural Model

In recent years, with the great success of pre-trained language models, the pre-trained BERT model has been gradually applied to the field of source code understanding. However, the time cost of training a language model from zero is very high, and how to transfer the pre-trained language model to the field of smart contract vulnerability detection is a hot research direction at present. In this paper, we propose a hybrid model to detect common vulnerabilities in smart contracts based on a lightweight pre-trained language model BERT and connected to a bidirectional gate recurrent unit model. The downstream neural network adopts the bidirectional gate recurrent unit neural network model with a hierarchical attention mechanism to mine more semantic features contained in the source code of smart contracts by using their characteristics. Our experiments show that our proposed hybrid neural network model SolBERT-BiGRU-Attention is fitted by a large number of data samples with smart contract vulnerabilities, and it is found that compared with the existing methods, the accuracy of our model can reach 93.85%, and the Micro-F1 Score is 94.02%.

An Intelligent Virtual Standard Patient for Medical Students Training Based on Oral Knowledge Graph

Virtual standard patient (VSP) is in high demand for medical students' diagnosis ability training in an efficient manner. Different from the traditional conversation system in medical dialogue generation, VSP needs a novel conversation paradigm to act as the patient instead of the doctor. However, existing conversation techniques still have limited ability in terms of generation of symptoms exhibited by patients with the personalized and knowledge-centered expressions. To alleviate these problems, we propose to construct a novel oral knowledge graph, which sufficiently provides medical clues of the certain disease. Accordingly, the VSP could accurately interact with the dentists for their underlying intention and express the symptoms characters in a natural style. To efficiently retrieve the related disease clues, the symptoms descriptions of the oral diseases are encoded into the oral knowledge graph, which could well organize the disease-centered symptom entities and speaking styles. Moreover, to transfer the common sense knowledge from existing large scale of medical knowledge graph to the specific oral knowledge graph, a coupled pre-trained Bert models is further designed to learn the related medical knowledge from coarse-level to fine-level hierarchically. Finally, a series of well-designed personalized templates are proposed to generate plausible and realistic answers in condition of the certain disease. We also conduct extensive user studies to demonstrate that the VSP satisfies the medical students' diagnosis practice requirement in terms of naturalness, realism, and topic relevance.

Detecting incongruity in the expression of emotions in short videos based on a multimodal approach

Every day people face uncertainty, which is already an integral part of their lives. Uncertainty creates risks for various kinds of companies, in particular, the financial sector may incur losses due to various kinds of human errors. People turn to the opinion of experts who have special knowledge to eliminate this uncertainty. It is established that the expert shows insolvency if he uses incongruent manipulation techniques. In this article we propose a method that allows solving the problem of congruence estimation. The hypothesis that a person with a prepared speech and a person with a spontaneous speech will have a different level of congruence is also put forward and tested in this work. The similarity of emotional states of verbal and nonverbal channels is evaluated in our solution for determining congruence. Convolutional neural networks (CNN) were used to assess a person’s emotional state from video and audio, speeth-to-text to extract the text of the speaker’s speech, and a pre-trained BERT model for subsequent analysis of emotional color. Tests have shown that with the help of this development it is possible not only to distinguish the incongruence of a person, but also to point out the unnatural nature of his origin (to distinguish a simply incongruent person from a deepfake).

The Comparison of Language Models with a Novel Text Filtering Approach for Turkish Sentiment Analysis

Today, comments can be made on many topics on web platforms with the development of the internet. Analyzing the data of these comments is essential for companies and data scientists. There are many methods for analyzing data. Recently, language models have also been used in many studies for sentiment analysis or text classification. In this study, Turkish sentiment analysis is performed using language models on hotel and movie review datasets. The language models are chosen because they are rarely used in Turkish literature. The pre-trained BERT, ALBERT, ELECTRA, and DistilBERT models for the Turkish language are trained and tested with these datasets. In addition, a text filtering method, which removes the words that can provide the opposition sentiment in the positive or negative labeled text, is proposed for sentiment analysis. These datasets obtained by this method are also retrained with language models and the accuracy values of their models are measured. The results of this study are compared with previous studies using the same datasets. As a result of the analysis, the accuracy values obtain state-of-the-art results with language models compared to previous studies. The best performance has been achieved by training the ELECTRA language model using the proposed text filtering method.

Claim Analyzer : Evaluating Credibility of Data

The current state of news in the era of computers The news environment in the modern computer age The news environment in the modern computer age The news environment in the modern computer age. The news environment in the modern computer age Social media platforms have taken the role of the antiquated traditional print media as a part of the news ecosystem in the current computer era. False news is spread at an astounding velocity and scale because social media platforms allow us to consume news much more rapidly and with less restricted editing. According to recent research, several efficient methods for spotting fake news encrypt social context-level data and news content sequential neural networks using a unidirectional examination of the text sequence. In order to represent the pertinent information of false news and improve classification performance while capturing semantic and long-distance connections in sentences, a bidirectional training strategy is a necessity. Claim Analyzer is only a model for evaluating the veracity of claims made online The statements might either be True or False. Fake news is a kind of propaganda in which false information is knowingly disseminated via news organizations and/or social media platforms. It is crucial to create methods of spotting false news information since its spread can have detrimental effects, such as influencing elections and widening political rifts. BERT is intended to simultaneously condition on both left and right context in all layers in order to pre-train deep bidirectional representations from unlabeled text. Therefore, state-of-the-art models for a variety of tasks, including question answering and language inference, may be created using just one extra output layer to fine-tune the pre-trained BERT model without making significant task-specific architecture adjustments. By merging several parallel blocks of the single-layer deep Convolutional Neural Network (CNN)with the BERT and variable kernel sizes and filters, we propose a BERT-based (Bidirectional Encoder Representations from Transformers) deep learning technique. The biggest obstacle to natural language understanding is ambiguity, which may be handled with the help of this combination. When applied to huge datasets, our suggested model provides88% accuracy.

Re-examining lexical and semantic attention: Dual-view graph convolutions enhanced BERT for academic paper rating

Automatically assessing academic papers has enormous potential to reduce peer-review burden and individual bias. Existing studies strive for building sophisticated deep neural networks to identify academic value based on comprehensive data, e.g., academic graphs and full papers. However, these data are not always easy to access. And the content of the paper rather than other features outside the paper should matter in a fair assessment. Furthermore, while BERT models can maintain general semantics by pre-training on large-scale corpora, they tend to be over-smoothing due to stacked self-attention layers among unfiltered input tokens. Therefore, it is nontrivial to figure out distinguishable value of an academic paper from its limited content. In this study, we propose a novel deep neural network, namely Dual-view Graph Convolutions Enhanced BERT (DGC-BERT), for academic paper acceptance estimation. We combine the title and abstract of the paper as input. Then, a pre-trained BERT model is employed to extract the paper’s general representations. Apart from hidden representations of the final layer, we highlight the first and last few layers as lexical and semantic views. In particular, we re-examine the dual-view filtered self-attention matrices via constructing two graphs, respectively. After that, two multi-hop Graph Convolutional Networks (GCNs) are separately employed to capture pivotal and distant dependencies between the tokens. Moreover, the dual-view representations are facilitated by each other with biaffine attention modules. And a re-weighting gate is proposed to further streamline the dual-view representations with the help of the original BERT representation. Finally, whether the submitted paper could be acceptable is predicted based on the original language model features cooperated with the dual-view dependencies. Extensive data analyses and the full paper based MHCNN studies provide insights into the task and structural functions. Comparison experiments on two benchmark datasets demonstrate that the proposed DGC-BERT significantly outperforms alternative approaches, especially the state-of-the-art models like MHCNN and BERT variants. Additional analyses reveal significance and explainability of the proposed modules in the DGC-BERT. Our codes and settings have been released on Github (https://github.com/ECNU-Text-Computing/DGC-BERT).

Informing the development of an outcome set and banks of items to measure mobility among individuals with acquired brain injury using natural language processing.

The sheer number of measures evaluating mobility and inconsistencies in terminology make it challenging to extract potential core domains and items. Automating a portion of the data synthesis would allow us to cover a much larger volume of studies and databases in a smaller fraction of the time compared to the usual process. Thus, the objective of this study was to identify a comprehensive outcome set and develop preliminary banks of items of mobility among individuals with acquired brain injury (ABI) using Natural Language Processing (NLP). An umbrella review of 47 reviews evaluating the content of mobility measures among individuals with ABI was conducted. A search was performed on 5 databases between 2000 and 2020. Two independent reviewers retrieved copies of the measures and extracted mobility domains and items. A pre-trained BERT model (state-of-the-art model for NLP) provided vector representations for each sentence. Using the International Classification of Functioning, Disability, and Health Framework (ICF) ontology as a guide for clustering, a k-means algorithm was used to retrieve clusters of similar sentences from their embeddings. The resulting embedding clusters were evaluated using the Silhouette score and fine-tuned according to expert input. The study identified 246 mobility measures, including 474 domains and 2109 items. Encoding the clusters using the ICF ontology and expert knowledge helped in regrouping the items in a way that is more closely related to mobility terminology. Our best results identified banks of items that were used to create a 24 comprehensive outcome sets of mobility, including Upper Extremity Mobility, Emotional Function, Balance, Motor Control, Self-care, Social Life and Relationships, Cognition, Walking, Postural Transition, Recreation, and Leisure Activities, Activities of Daily Living, Physical Functioning, Communication, Work/Study, Climbing, Sensory Functions, General Health, Fatigue, Functional Independence, Pain, Alcohol and Drugs Use, Transportation, Sleeping, and Finances. The banks of items of mobility domains represent a first step toward establishing a comprehensive outcome set and a common language of mobility to develop the ontology. It enables researchers and healthcare professionals to begin exposing the content of mobility measures as a way to assess mobility comprehensively.

Applying BERT for Early-Stage Recognition of Persistence in Chat-Based Social Engineering Attacks

Chat-based social engineering (CSE) attacks are attracting increasing attention in the Small-Medium Enterprise (SME) environment, given the ease and potential impact of such an attack. During a CSE attack, malicious users will repeatedly use linguistic tricks to eventually deceive their victims. Thus, to protect SME users, it would be beneficial to have a cyber-defense mechanism able to detect persistent interlocutors who repeatedly bring up critical topics that could lead to sensitive data exposure. We build a natural language processing model, called CSE-PersistenceBERT, for paraphrase detection to recognize persistency as a social engineering attacker’s behavior during a chat-based dialogue. The CSE-PersistenceBERT model consists of a pre-trained BERT model fine-tuned using our handcrafted CSE-Persistence corpus; a corpus appropriately annotated for the specific downstream task of paraphrase recognition. The model identifies the linguistic relationship between the sentences uttered during the dialogue and exposes the malicious intent of the attacker. The results are satisfactory and prove the efficiency of CSE-PersistenceBERT as a recognition mechanism of a social engineer’s persistent behavior during a CSE attack.

Identifying Cantonese rumors with discriminative feature integration in online social networks

To reduce the negative impacts of rumors on the real world, rumor detection on social networks has practical significance. Currently, the research on Chinese rumor detection is relatively comprehensive, but Cantonese rumors are much less investigated. As a main dialect of Chinese, Cantonese has more than 60 million speakers globally, but there are some great challenges in the Cantonese rumor detection. Firstly, there is no available benchmark dataset of Cantonese rumors. Secondly, it is a significant challenge to learn the unique linguistic characteristics of Cantonese. Thirdly, traditional rumor detection approaches cannot be directly applied to Cantonese rumors. Therefore, we propose a novel framework for Cantonese rumor detection using deep neural networks with feature fusion. To the best of our knowledge, it is the first study conducted on Cantonese rumor detection on social networks. Specifically, we build a Cantonese rumor dataset and a multi-domain Cantonese corpus. Next, a total of 27 statistical features are extracted and seven of them are newly proposed. Then, a novel deep learning model called BLA is designed to identify Cantonese rumors, which generates text and Jyutping embeddings using a further pre-trained BERT model and a CNN model. Moreover, the BLA model integrates the statistical and semantic features to implement the classification of Cantonese rumors. Experiments demonstrate that the BLA model achieves a remarkable Cantonese rumor detection performance with an F1 Score of 0.9225.

MLM-based typographical error correction of unstructured medical texts for named entity recognition

BackgroundUnstructured text in medical records, such as Electronic Health Records, contain an enormous amount of valuable information for research; however, it is difficult to extract and structure important information because of frequent typographical errors. Therefore, improving the quality of data with errors for text analysis is an essential task. To date, few prior studies have been conducted addressing this. Here, we propose a new methodology for extracting important information from unstructured medical texts by overcoming the typographical problem in surgical pathology records related to lung cancer.MethodsWe propose a typo correction model that considers context, based on the Masked Language Model, to solve the problem of typographical errors in real-world medical data. In addition, a word dictionary was used for the typo correction model based on PubMed abstracts. After refining the data through typo correction, fine tuning was performed on pre-trained BERT model. Next, deep learning-based Named Entity Recognition (NER) was performed. By solving the quality problem of medical data, we sought to improve the accuracy of information extraction in unstructured text data.ResultsWe compared the performance of the proposed typo correction model based on contextual information with an existing SymSpell model. We confirmed that our proposed model outperformed the existing model in a typographical correction task. The F1-score of the model improved by approximately 5% and 9% when compared with the model without contextual information in the NCBI-disease and surgical pathology record datasets, respectively. In addition, the F1-score of NER after typo correction increased by 2% in the NCBI-disease dataset. There was a significant performance difference of approximately 25% between the before and after typo correction in the Surgical pathology record dataset. This confirmed that typos influenced the information extraction of the unstructured text.ConclusionWe verified that typographical errors in unstructured text negatively affect the performance of natural language processing tasks. The proposed method of a typo correction model outperformed the existing SymSpell model. This study shows that the proposed model is robust and can be applied in real-world environments by focusing on the typos that cause difficulties in analyzing unstructured medical text.

When BERT Started Traveling: TourBERT—A Natural Language Processing Model for the Travel Industry

In recent years, Natural Language Processing (NLP) has become increasingly important for extracting new insights from unstructured text data, and pre-trained language models now have the ability to perform state-of-the-art tasks like topic modeling, text classification, or sentiment analysis. Currently, BERT is the most widespread and widely used model, but it has been shown that a potential to optimize BERT can be applied to domain-specific contexts. While a number of BERT models that improve downstream tasks’ performance for other domains already exist, an optimized BERT model for tourism has yet to be revealed. This study thus aimed to develop and evaluate TourBERT, a pre-trained BERT model for the tourism industry. It was trained from scratch and outperforms BERT-Base in all tourism-specific evaluations. Therefore, this study makes an essential contribution to the growing importance of NLP in tourism by providing an open-source BERT model adapted to tourism requirements and particularities.

Domain-Adaptive Pre-training BERT Model for Test and Identification Domain NER Task

Name entity recognition (NER) is the foundation of a wide range of natural language processing (NLP) task in the domain of test identification. In this paper, we continue to train the pre-trained BERT model by unlabeled texts related to the domain of test identification, so as to inject domain knowledge into the pre-trained BERT model and realize the domain adaptation. The experiment results show that the proposed domain-adaptive pre-training method increases the F1 value by 1% compared with the baseline in the domain of test identification NER task.

CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search

Neural networks (NNs) have been successfully deployed in various applications of artificial intelligence. However, architectural design of NNs is still a challenging problem. This is due to the need to navigate a search space based on a large number of hyperparameters. This forces the search space of possible architectures to grow exponentially. Using a trial-and-error design approach is very time consuming and leads to suboptimal architectures. In addition, approaches, such as neural architecture search based on reinforcement learning and differentiable gradient-based architecture search, often incur huge computational costs or significant memory requirements. To address these challenges, we propose the CURIOUS NN synthesis methodology. It uses a performance predictor to efficiently navigate the architectural search space with an evolutionary search process. The predictor is built using quasi Monte-Carlo sampling, boosted decision tree regression, and an intelligent iterative sampling method. It is designed to be sample efficient. CURIOUS starts from a base architecture and explores the architectural search space to obtain a variant of the base architecture with the highest performance. This search framework is general and covers all important NN architecture types, e.g., feedforward NNs (FFNNs), convolutional NNs (CNNs), recurrent NNs (RNNs), and transformers. We evaluate the performance of CURIOUS on various datasets and base architectures. Through these experiments, we demonstrate significant performance improvements over the baseline architectures. For the MNIST dataset, our CNN architecture achieves an error rate of 0.66%, with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.6\times $ </tex-math></inline-formula> fewer parameters compared to the LeNet-5 baseline. For the CIFAR-10 dataset, we use the ResNet architectures and residual networks with Shake-Shake regularization as the baselines. Our synthesized ResNet-18 has a 2.52% accuracy improvement over the original ResNet-18, 1.74% over ResNet-101, and 0.16% over ResNet-1001, while requiring comparable number of parameters and floating-point operations to the original ResNet-18. This result shows that instead of just increasing the number of layers to increase accuracy, an alternative is to use a better NN architecture with a small number of layers. In addition, CURIOUS achieves an error rate of just 2.69% with a variant of the residual architecture with Shake-Shake regularization. We also use the set of optimized hyperparameters found for ResNet-18 on the CIFAR-10 dataset to train and evaluate the model on the ImageNet dataset, and show 3.43% (1.83%) improvement in the top-1 (top-5) error rate compared to the original ResNet-18 model. CURIOUS also obtains the highest accuracy for various other FFNNs that are geared toward edge devices and IoT sensors. In addition, we use CURIOUS to search for deep RNN architectures for the SICK dataset for sentence similarity evaluation. It achieves a mean-squared error of only 0.2060, improving upon the base network performance, without the need to stack multiple long short-term memories. We also use CURIOUS to search for a better NN classifier for the sentiment analysis task on the Stanford sentiment treebank dataset using a pretrained BERT model and again demonstrate improvements in performance.

SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting

SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting

MolRoPE-BERT: An enhanced molecular representation with Rotary Position Embedding for molecular property prediction

Molecular property prediction is a significant task in drug discovery. Most deep learning-based computational methods either develop unique chemical representation or combine complex model. However, researchers are less concerned with the possible advantages of enormous quantities of unlabeled molecular data. Since the obvious limited amount of labeled data available, this task becomes more difficult. In some senses, SMILES of the drug molecule may be regarded of as a language for chemistry, taking inspiration from natural language processing research and current advances in pretrained models. In this paper, we incorporated Rotary Position Embedding(RoPE) efficiently encode the position information of SMILES sequences, ultimately enhancing the capability of the BERT pretrained model to extract potential molecular substructure information for molecular property prediction. We proposed the MolRoPE-BERT framework, an new end-to-end deep learning framework that integrates an efficient position coding approach for capturing sequence position information with a pretrained BERT model for molecular property prediction. To generate useful molecular substructure embeddings, we first exclusively train the MolRoPE-BERT on four million unlabeled drug SMILES(i.e., ZINC 15 and ChEMBL 27). Then, we conduct a series of experiments to evaluate the performance of our proposed MolRoPE-BERT on four well-studied datasets. Compared with conventional and state-of-the-art baselines, our experiment demonstrated comparable or superior performance.

A Use Case of Patent Classification Using Deep Learning with Transfer Learning

Abstract Purpose Patent classification is one of the areas in Intellectual Property Analytics (IPA), and a growing use case since the number of patent applications has been increasing worldwide. We propose using machine learning algorithms to classify Portuguese patents and evaluate the performance of transfer learning methodologies to solve this task. Design/methodology/approach We applied three different approaches in this paper. First, we used a dataset available by INPI to explore traditional machine learning algorithms and ensemble methods. After preprocessing data by applying TF-IDF, FastText and Doc2Vec, the models were evaluated by cross-validation in 5 folds. In a second approach, we used two different Neural Networks architectures, a Convolutional Neural Network (CNN) and a bi-directional Long Short-Term Memory (BiLSTM). Finally, we used pre-trained BERT, DistilBERT, and ULMFiT models in the third approach. Findings BERTTimbau, a BERT architecture model pre-trained on a large Portuguese corpus, presented the best results for the task, even though with a performance of only 4% superior to a LinearSVC model using TF-IDF feature engineering. Research limitations The dataset was highly imbalanced, as usual in patent applications, so the classes with the lowest samples were expected to present the worst performance. That result happened in some cases, especially in classes with less than 60 training samples. Practical implications Patent classification is challenging because of the hierarchical classification system, the context overlap, and the underrepresentation of the classes. However, the final model presented an acceptable performance given the size of the dataset and the task complexity. This model can support the decision and improve the time by proposing a category in the second level of ICP, which is one of the critical phases of the grant patent process. Originality/value To our knowledge, the proposed models were never implemented for Portuguese patent classification.

Implementasi Deteksi Judul Berita Clickbait Berbahasa Indonesia dengan pre-trained model Multilingual BERT Pada Aplikasi Berbasis Chrome Extension

Clickbait news title is often used by online news portal. The purpose of clickbait is to attract reader to open and read the news. Furthermore, news containing clickbait title can give negative impact by reducing the essence of important news. Therefore, clickbait detection tool is needed to avoid the clickbait news title. Chrome extension was chosen in this study because it supports all Chrome based browsers, such as Google Chrome, Chromium, Microsoft Edge, and Opera so that many users apply this program. In this study, Chrome extension-based application was designed and integrated by using artificial intelligence model. This application also utilized the availability of pre-trained multilingual BERT model as Natural Language Processing (NLP) which will be used to predict a clickbait news title. This study used Multilingual BERT model as NLP because this model has been trained into 104 languages, including Bahasa Indonesia and it has significant performance. The result of this study can detect clickbait news along with 92% of AUC-ROC value.

RECA: Relation Extraction Based on Cross-Attention Neural Network

Extracting entities and relations, as a crucial part of many tasks in natural language processing, transforms the unstructured text information into structured information and provides corresponding data support for knowledge graph (KG) and knowledge vault (KV) construction. Nevertheless, the mainstream relation-extraction methods, the pipeline method and the joint method, ignore the dependency between the subject entity and the object entity. This work introduces a pre-trained BERT model and a dilated gated convolutional neural network (DGCNN) as an encoder to distinguish the long-range semantics representation from the input sequence. In addition, we propose a cross-attention neural network as a decoder to learn the importance of each subject word for each word of the input sequence. Experiments were undertaken with two extensive datasets, the New York Times Corpus (NYT) and WebNLG Corpus, and showed that our model performs significantly better than the CasRel model, outperforming the baseline by 1.9% and 0.7% absolute gain in terms of F1-score.

Do We Need a Specific Corpus and Multiple High-Performance GPUs for Training the BERT Model? An Experiment on COVID-19 Dataset

The COVID-19 pandemic has impacted daily lives around the globe. Since 2019, the amount of literature focusing on COVID-19 has risen exponentially. However, it is almost impossible for humans to read all of the studies and classify them. This article proposes a method of making an unsupervised model called a zero-shot classification model, based on the pre-trained BERT model. We used the CORD-19 dataset in conjunction with the LitCovid database to construct new vocabulary and prepare the test dataset. For NLI downstream task, we used three corpora: SNLI, MultiNLI, and MedNLI. We significantly reduced the training time by 98.2639% to build a task-specific machine learning model, using only one Nvidia Tesla V100. The final model can run faster and use fewer resources than its comparators. It has an accuracy of 27.84%, which is lower than the best-achieved accuracy by 6.73%, but it is comparable. Finally, we identified that the tokenizer and vocabulary more specific to COVID-19 could not outperform the generalized ones. Additionally, it was found that BART architecture affects the classification results.