BERT Language Model Research Articles

Topic modelling as a method for framing analysis of news coverage of the Russia-Ukraine war in 2022–2023

This study critically analyses the representation of the Russia-Ukraine war in Western (the Euronews) and Eastern (the Kyiv Post) media discourses. It examines how media organisations shape narratives through strategic framing. Employing the Natural Language Processing technique – Topic Modelling – with a generative probabilistic model LDA and a transformer-based language model BERT, the study reveals generic frames elaborated by more specific extensions, shedding light on media portrayal of economy, public opinion, security & defence, external regulations, policy evaluation, and health & safety sectors. Through Named Entity Recognition with roBERTa, Sentiment Analysis with distilBERT, and Corpus Linguistics methods with LancsBox X, interpretation of these overarching frames provides a comprehensive analysis of the nuances in narratives, societal perceptions and policy decisions amidst the ongoing war.

Addressing data sparsity and cold-start challenges in recommender systems using advanced deep learning and self-supervised learning techniques

ABSTRACT The efficacy of e-commerce conversion rates relies on precise and personalized product recommendations within recommendation systems (RS). While collaborative filtering-based RS has demonstrated success, challenges such as sparsity and cold-start issues in the user-item matrix can impede optimal functionality. To address these challenges, there is a need to integrate additional information sources, encompassing item/user profiles and textual reviews. This study introduces an innovative RS architecture that seamlessly combines self-supervised learning (SSL) and collaborative filtering techniques with BERT-DNN to surmount these obstacles. The distinctiveness of our approach lies in integrating self-supervised learning with collaborative filtering and contextualized data obtained from BERT-DNN, providing a profound understanding of item profiles to enhance comprehension of user preferences and item characteristics. This refined understanding, operational in conjunction with collaborative filtering models like ItemKNN and UserKNN, empowers the system to generate highly personalized recommendations. The proposed method entails several pivotal steps: developing the BERT language model for textual embeddings in item profiles, conducting dimensionality reduction, constructing a Deep Neural Network, implementing self-supervised learning with both UserKNN and ItemKNN CF methods, and employing an ensemble learning technique. Empirical results substantiate the efficacy of our approach, with a specific focus on the innovative fusion of BERT-DNN with self-supervised learning and KNN CF methodologies, showcasing substantial improvements across diverse performance metrics. This underscores the practical importance of leveraging contextualized BERT-DNN data, strengthening the recommendation mechanism, and ultimately enhancing the overall performance of the RS.

Enhancing the design of voting advice applications with BERT language model.

The relevance and importance of voting advice applications (VAAs) are demonstrated by their popularity among potential voters. On average, around 30% of voters take into account the recommendations of these applications during elections. The comparison between potential voters' and parties' positions is made on the basis of VAA policy statements on which users are asked to express opinions. VAA designers devote substantial time and effort to analyzing domestic and international politics to formulate policy statements and select those to be included in the application. This procedure involves manually reading and evaluating a large volume of publicly available data, primarily party manifestos. A problematic part of the work is the limited time frame. This study proposes a system to assist VAA designers in formulating, revising, and selecting policy statements. Using pre-trained language models and machine learning methods to process politics-related textual data, the system produces a set of suggestions corresponding to relevant VAA statements. Experiments were conducted using party manifestos and YouTube comments from Japan, combined with VAA policy statements from six Japanese and two European VAAs. The technical approaches used in the system are based on the BERT language model, which is known for its capability to capture the context of words in the documents. Although the output of the system does not completely eliminate the need for manual human assessment, it provides valuable suggestions for updating VAA policy statements on an objective, i.e., bias-free, basis.

머신러닝 기반 색채적 특성 분석 및 감성 분류에 관한 연구

This study focuses on developing a methodology for classifying tourist destination images based on their color characteristics and emotional categories using machine learning algorithms. By analyzing user-generated content collected through digital media and smartphones, particularly tourist destination images from Instagram, the research examines how the initial impressions and imagery of tourist destinations impact tourists' emotions. Color and text data were analyzed using machine learning algorithms and the BERT language model to interpret the color characteristics and emotional messages conveyed by these images. The research demonstrates that analyzing and classifying the color characteristics and emotions of tourist destination images significantly influence tourists' choices and emotions. It shows that considering these two factors in an interconnected manner, rather than in isolation, enhances their utility. Specifically, color characteristics are crucial elements that can improve the attractiveness of tourist destination images. This study reveals that these images can significantly affect tourists' emotions and experiences, offering new insights and effective strategies for optimizing the tourism experience. Moreover, by systematically analyzing and utilizing the data, it is possible to provide customized suggestions that reflect the preferred images of tourist destinations and user tendencies. This research provides critical insights into image management and marketing strategies for tourist destinations, helping to attract tourists' interest and create positive tourism experiences. Ultimately, the study aims to deepen the understanding of the tourism industry through the color characteristics and emotional classification of tourist destination images, making significant contributions to the enhancement of image management and marketing strategies for tourist destinations.

ShallowBKGC: a BERT-enhanced shallow neural network model for knowledge graph completion

Knowledge graph completion aims to predict missing relations between entities in a knowledge graph. One of the effective ways for knowledge graph completion is knowledge graph embedding. However, existing embedding methods usually focus on developing deeper and more complex neural networks, or leveraging additional information, which inevitably increases computational complexity and is unfriendly to real-time applications. In this article, we propose an effective BERT-enhanced shallow neural network model for knowledge graph completion named ShallowBKGC. Specifically, given an entity pair, we first apply the pre-trained language model BERT to extract text features of head and tail entities. At the same time, we use the embedding layer to extract structure features of head and tail entities. Then the text and structure features are integrated into one entity-pair representation via average operation followed by a non-linear transformation. Finally, based on the entity-pair representation, we calculate probability of each relation through multi-label modeling to predict relations for the given entity pair. Experimental results on three benchmark datasets show that our model achieves a superior performance in comparison with baseline methods. The source code of this article can be obtained from https://github.com/Joni-gogogo/ShallowBKGC.

Enhancing abusive language detection: A domain-adapted approach leveraging BERT pre-training tasks

The widespread adoption of deep learning approaches in natural language processing is largely attributed to their exceptional performance across diverse tasks. Notably, Transformer-based models, such as BERT, have gained popularity for their remarkable efficacy and their ease of adaptation (via fine-tuning) across various domains. Despite their success, fine-tuning these models for informal language, particularly instances involving offensive expressions, presents a major challenge due to limitations in vocabulary coverage and contextual information for such tasks. To address these challenges, we propose the domain adaptation of the BERT language model for the task of detecting abusive language. Our approach involves constraining the language model with the adaptation and paradigm shift of two default pre-trained tasks, the design of two datasets specifically engineered to support the adapted pre-training tasks, and the proposal of a dynamic weighting loss function. The evaluation of these adapted configurations on six datasets dedicated to abusive language detection reveals promising outcomes, with a significant enhancement observed compared to the base model. Furthermore, our proposed methods yield competitive results when compared to state-of-the-art approaches, establishing a robust and easily trainable model for the effective identification of abusive language.

Odio-BERT: Evaluating domain task impact in hate speech detection

The rise of social media and micro-blogging platforms has led to concerns about hate speech, its potential to incite violence, psychological trauma, extremist beliefs, and self-harm. We have proposed a novel model, Odio-BERT for detecting hate speech using a pretrained BERT language model. This specialized model is specifically designed for detecting hate speech in the Spanish language, and when compared to existing models, it consistently outperforms them. The study provides valuable insights into addressing hate speech in the Spanish language and explores the impact of domain tasks.

Abstract 3475: Prediction of nausea or vomiting, and fatigue or malaise in cancer care

Abstract Introduction: Natural language processing has recently achieved unprecedented performances for several medical tasks, but requires additional improvements in oncology. Moreover, very few projects have assessed the potential of language models to help prevent the most frequent serious or severe medical events in medical oncology. We aim at predicting nausea or vomiting (ICD10 code R11), and fatigue or malaise (ICD10 code R53) from patients' medical reports. Material and methods: The study included all the patients of Centre Léon Bérard between 2000 and 2023 that have not refused to share their data for this analysis. We have retrieved all the clinical notes and manual coding of hospitalization stays in ICD10, in French. We have pretrained a BERT language model with a masking strategy on this data and then fine-tuned it and compared it to several medical pretrained open source models (DrBert and K-memBERT). The labels were medical events leading to or associated with a hospitalization in the 90 days after every patients’ notes. For OncoBERT, we included sequential reports from the patients’ history, along with a time-encoding layer, and integrated it in a final transformer layer. Results: We analyzed 140,523 patients, representing 2,515,957 pseudo-anonymized text reports and 6.6M hospitalizations codes in total. The medical texts were consultations reports (56%), end-of-stay reports (17%) and hospitalization summaries (9%). The most frequent types of oncology treatments received by the patients at each time point were 18.8% for chemotherapy, 10.8% for targeted therapies and 1.3% for immunotherapies. The most frequent medical events were nausea or vomiting (20% of patients with 1 or more events), and fatigue or malaise (18% of patients with 1 or more events). In the final dataset, nausea and vomiting (R11) accounted for 16% of the labels while malaise and fatigue (R53) accounted for 24.7% of the labels. We performed random undersampling of reports without any event to balance the label dataset. The fine-tuning on R11 and R53 achieved the performances of 0.58 macro-aucpr (OncoBERT) and 0.50 macro-aucpr (best open-source model) on the validation set. Conclusion: The language models achieved high performance on the prediction of the most frequent serious medical events in our hospital dedicated to cancer care. We plan to validate the external performances of these models on collaborating hospitals and prospectively and improve the interpretations that we will present at the congress. Citation Format: Raphael Vienne, Quentin Filori, Vincent Susplugas, Hugo Crochet, Loic Verlingue. Prediction of nausea or vomiting, and fatigue or malaise in cancer care [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3475.

Development of a Language Model for Named-Entity-Recognition in Aerospace Requirements

We address the challenges inherent in converting natural language (NL) requirements into machine-readable formats by investigating the application of named-entity recognition (NER) within the aerospace domain. Recognizing the necessity for domain-specific language models, we developed an open-source annotated aerospace corpus and fine-tuned different versions of the BERT language model on the corpus to create aeroBERT-NER: a new model for identifying named entities (NEs) in the aerospace domain. A comparison between aeroBERT-NER and BERTBASE-NER demonstrated the superior performance of aeroBERT-NER in identifying NEs within a set of aerospace requirements. The identified NEs contribute to the development of a glossary, promoting consistent terminology usage in aerospace requirements and addressing challenges associated with the standardization of NL requirements.

From coarse to fine: Enhancing multi-document summarization with multi-granularity relationship-based extractor

Multi-Document Summarization (MDS) is a challenging task due to the fact that multiple documents not only have extremely long inputs but may also be overlapping, complementary, or contradictory to each other. In this paper, we propose to capture complex cross-document interactions to handle lengthy inputs for better multi-document summarization. Specifically, we present MDS-MGRE, a coarse-to-fine MDS framework that introduces Multi-Granularity Relationships into an Extract-then-summarize pipeline. In the coarse-grained stage, multi-granularity embedding, heterogeneous graph construction, and MGRExtractor work together to convert redundant multi-documents into compact meta-documents. We first utilize pre-trained language model BERT to obtain semantically rich embeddings for documents at different granularities, including documents, paragraphs, sentence-sets, and sentences. Then, we construct a heterogeneous graph with 4 types of nodes (document nodes, paragraph nodes, sentence-set nodes, and sentence nodes) and corresponding connecting edges to model rich document relationships. Furthermore, we propose a novel Multi-Granularity Relationship-based Extractor (MGRExtractor) to produce meta-documents by efficiently pruning heterogeneous graphs. More precisely, it consists of 4 main modules: noise removal, redundancy removal, multi-granularity scoring, and sentence-set selection. In the fine-grained stage, we employ the large configuration of BART as our abstractive summarizer to generate system summaries from the extracted meta-documents. Experimental results on two benchmark datasets show that our framework significantly outperforms strong baselines with comparable parameters, and slightly underperforms methods with a maximum encoding length of 16,384 tokens. For Multi-News and WCEP, automatic evaluation results show that MDS-MGRE achieves an average performance improvement of 1.75% and 8.77% compared to the state-of-the-art systems with comparable parameters, respectively. Such positive results demonstrate the benefits of generating high-quality meta-documents to enhance MDS by modeling rich document relationships.

Comparison of an Ensemble of Machine Learning Models and the BERT Language Model for Analysis of Text Descriptions of Brain CT Reports to Determine the Presence of Intracranial Hemorrhage.

Seven machine learning algorithms and three text vectorization techniques were selected as models to solve the binary classification problem. These models were trained on textual data represented by 3980 brain CT reports from 56 inpatient medical facilities in Moscow. The study utilized three text vectorization techniques: bag of words, TF-IDF, and word2vec. The resulting data were then processed by the following machine learning algorithms: decision tree, random forest, logistic regression, nearest neighbors, support vector machines, Catboost, and XGboost. Data analysis and pre-processing were performed using NLTK (Natural Language Toolkit, version 3.6.5), libraries for character-based and statistical processing of natural language, and Scikit-learn (version 0.24.2), a library for machine learning containing tools to tackle classification challenges. MedRuBertTiny2 was taken as a BERT transformer model pre-trained on medical data. Based on the training and testing outcomes from seven machine learning algorithms, the authors selected three algorithms that yielded the highest metrics (i.e. sensitivity and specificity): CatBoost, logistic regression, and nearest neighbors. The highest metrics were achieved by the bag of words technique. These algorithms were assembled into an ensemble using the stacking technique. The sensitivity and specificity for the validation dataset separated from the original sample were 0.93 and 0.90, respectively. Next, the ensemble and the BERT model were trained on an independent dataset containing 9393 textual radiology reports also divided into training and test sets. Once the ensemble was tested on this dataset, the resulting sensitivity and specificity were 0.92 and 0.90, respectively. The BERT model tested on these data demonstrated a sensitivity of 0.97 and a specificity of 0.90. When analyzing textual reports of brain CT scans with signs of intracranial hemorrhage, the trained ensemble demonstrated high accuracy metrics. Still, manual quality control of the results is required during its application. The pre-trained BERT transformer model, additionally trained on diagnostic textual reports, demonstrated higher accuracy metrics (p<0.05). The results show promise in terms of finding specific values for both binary classification task and in-depth analysis of unstructured medical information.

A Natural Language Processing Approach Towards Harmonized Communication of Uncertainties Identified During the European Medicine Authorization Process.

Within the European Union, the European Medicines Agency's (EMA's) European Public Assessment Report (EPAR) is an important source of information for healthcare professionals and patients that allows them to understand important risks and uncertainties associated with the use of a medicine. However, the EPAR sections describing such important uncertainties can differ substantially in wording, length, and detail, thereby potentially limiting understanding. In this study, we therefore present a natural language processing approach to cluster sentences extracted from the sections on uncertainties in EPARs of centrally authorized medicines, as a steppingstone to harmonization of text describing uncertainties. We used a BERT language model together with dimensionality reduction (Uniform Manifold Approximation and Projection (UMAP)) and clustering (Density-Based Spatial Clustering of Applications with Noise (DBSCAN)) to identify semantic similarities between sentences. Clusters were labeled according to an overarching topic by reviewing the semantically similar sentences. Each cluster was also characterized according to medicine-related characteristics, such as efficacy or side effects. In total, 1,648 medicines were included in this study. For 573 of these medicines (authorized July 27, 2010 to December 31, 2022), we identified an EPAR that described a complete regulatory dossier and contained sections on uncertainties. Of these, 553 EPARs could be attributed to unique active substance-indication combinations. In these 553 EPARs, we identified 13,105 sentences in sections on uncertainties, leading to 26 clusters of which 2 were labeled as noise. The clusters and associated topics provided in this article can be used by regulators and medicine developers as a steppingstone toward a unified way of communicating uncertainties identified during the EMA process to the broader public.

Research on 5G User Perception Detection and Experience Improvement Optimization Based on Capsule Network

COVID-19 caused a global public disaster as well as an economic crisis, and other challenges. The fifth-generation network, or 5G, connects practically every machine, person, and thing on the planet. We can analyse the public's opinions and sentiments connected to COVID-19 from 5G user-generated content on social media, which will eventually aid in promoting health intervention strategies and designing successful projects based on public perceptions. The BERT language model is first used to preprocess data that has been obtained from Sina Weibo. Following that, the features of the preprocessed data are chosen using a class-wise information technique. Finally, a capsule network (CapsNet) approach is used to identify the 5G user perception and experience optimization. Dynamic routing algorithm is used for optimizing the capsule network. By comparing the suggested framework's performance with certain existing approaches, its effectiveness is evaluated. Simulation results show that the proposed method is more accurate than previous approaches at identifying 5G user experiences.

Prediction of linear B-cell epitopes based on protein sequence features and BERT embeddings

Linear B-cell epitopes (BCEs) play a key role in the development of peptide vaccines and immunodiagnostic reagents. Therefore, the accurate identification of linear BCEs is of great importance in the prevention of infectious diseases and the diagnosis of related diseases. The experimental methods used to identify BCEs are both expensive and time-consuming and they do not meet the demand for identification of large-scale protein sequence data. As a result, there is a need to develop an efficient and accurate computational method to rapidly identify linear BCE sequences. In this work, we developed the new linear BCE prediction method LBCE-BERT. This method is based on peptide chain sequence information and natural language model BERT embedding information, using an XGBoost classifier. The models were trained on three benchmark datasets. The model was training on three benchmark datasets for hyperparameter selection and was subsequently evaluated on several test datasets. The result indicate that our proposed method outperforms others in terms of AUROC and accuracy. The LBCE-BERT model is publicly available at: https://github.com/Lfang111/LBCE-BERT.

Causality and signalling of garden-path sentences.

Sheaves are mathematical objects that describe the globally compatible data associated with open sets of a topological space. Original examples of sheaves were continuous functions; later they also became powerful tools in algebraic geometry, as well as logic and set theory. More recently, sheaves have been applied to the theory of contextuality in quantum mechanics. Whenever the local data are not necessarily compatible, sheaves are replaced by the simpler setting of presheaves. In previous work, we used presheaves to model lexically ambiguous phrases in natural language and identified the order of their disambiguation. In the work presented here, we model syntactic ambiguities and study a phenomenon in human parsing called garden-pathing. It has been shown that the information-theoretic quantity known as 'surprisal' correlates with human reading times in natural language but fails to do so in garden-path sentences. We compute the degree of signalling in our presheaves using probabilities from the large language model BERT and evaluate predictions on two psycholinguistic datasets. Our degree of signalling outperforms surprisal in two ways: (i) it distinguishes between hard and easy garden-path sentences (with a [Formula: see text]-value [Formula: see text]), whereas existing work could not, (ii) its garden-path effect is larger in one of the datasets (32 ms versus 8.75 ms per word), leading to better prediction accuracies. This article is part of the theme issue 'Quantum contextuality, causality and freedom of choice'.

How do control tokens affect natural language generation tasks like text simplification

Abstract Recent work on text simplification has focused on the use of control tokens to further the state-of-the-art. However, it is not easy to further improve without an in-depth comprehension of the mechanisms underlying control tokens. One unexplored factor is the tokenization strategy, which we also explore. In this paper, we (1) reimplemented AudienCe-CEntric Sentence Simplification, (2) explored the effects and interactions of varying control tokens, (3) tested the influences of different tokenization strategies, (4) demonstrated how separate control tokens affect performance and (5) proposed new methods to predict the value of control tokens. We show variations of performance in the four control tokens separately. We also uncover how the design of control tokens could influence performance and give some suggestions for designing control tokens. We show the newly proposed method with higher performance in both SARI (a common scoring metric in text simplificaiton) and BERTScore (a score derived from the BERT language model) and potential in real applications.

Heterogeneous data-based information retrieval using a fine-tuned pre-trained BERT language model

Heterogeneous data-based information retrieval using a fine-tuned pre-trained BERT language model

A full-document analysis of the semantic relation between European Public Assessment Reports and EMA guidelines using a BERT language model

In the European Union, the Committee for Medicinal Products for Human Use of the European Medicines Agency (EMA) develop guidelines to guide drug development, supporting development of efficacious and safe medicines. A European Public Assessment Report (EPAR) is published for every medicine application that has been granted or refused marketing authorisation within the EU. In this work, we study the use of text embeddings and similarity metrics to investigate the semantic similarity between EPARs and EMA guidelines. All 1024 EPARs for initial marketing authorisations from 2008 to 2022 was compared to the 669 current EMA scientific guidelines. Documents were converted to plain text and split into overlapping chunks, generating 265,757 EPAR and 27,649 guideline text chunks. Using a Sentence BERT language model, the chunks were transformed into embeddings and fed into an in-house piecewise matching algorithm to estimate the full-document semantic distance. In an analysis of the document distance scores and product characteristics using a linear regression model, EPARs of anti-virals for systemic use (ATC code J05) and antihemorrhagic medicines (B02) present with statistically significant lower overall semantic distance to guidelines compared to other therapeutic areas, also when adjusting for product age and EPAR length. In conclusion, we believe our approach provides meaningful insight into the interplay between EMA scientific guidelines and the assessment made during regulatory review, and could potentially be used to answer more specific questions such as which therapeutic areas could benefit from additional regulatory guidance.

Dimensions of constructional meanings in the German Constructicon: Why collo-profiles matter

Abstract A constructicon, i.e., a structured inventory of constructions, essentially aims at documenting functions of lexical and grammatical constructions. Among other parameters, so-called constructional collo-profiles, as introduced by Herbst (2018, 2020), are conclusive for determining constructional meanings. They provide information on how relevant individual words are for construction slots, they hint at usage preferences of constructions and serve as a helpful indicator for semantic peculiarities of constructions. However, even though collo-profiles constitute an indispensable component of constructicon entries, they pose major challengers for constructicographers: For a constructicographic enterprise it is not feasible to conduct collostructional analyses for hundreds or even thousands of constructions. In this article, we introduce a procedure based on the large language model BERT that allows to predict collo-profiles without having to extensively annotate instances of constructions in a given corpus. Specifically, by discussing the constructions X macht Y ADJP (‘x makes Y ADJ’, e.g. he drives him crazy) and N1 PREP N1 (e.g., bumper to bumper, constructions over constructions), we show how the developed automated system generates collo-profiles based on a limited number of annotated instances. Finally, we place collo-profiles alongside other dimensions of constructional meanings included in the German Constructicon.

Predictive Recognition of DNA-binding Proteins Based on Pre-trained Language Model BERT.

Identifying proteins is crucial for disease diagnosis and treatment. With the increase of known proteins, large-scale batch predictions are essential. However, traditional biological experiments being time-consuming and expensive are difficult to accomplish this task efficiently. Nevertheless, deep learning algorithms based on big data analysis have manifested potential in this aspect. In recent years, language representation models, especially BERT, have made significant advancements in natural language processing. In this paper, using three protein segmentation methods and three encoder numbers, nine BERT models with different sizes are constructed to predict whether known proteins are DNA-binding proteins or not. Furthermore, based on the concept of protein motifs, multi-scale convolutional networks are fused into the models to extract the local features of DNA-binding proteins. Finally, we find that the larger the number of encoders, the better the model predictions under the condition of considering each amino acid in the protein as a word. Our proposed algorithm achieves 81.88% sensitivity and 0.39 MCC value on the test set. Furthermore, it achieves 62.41% accuracy on the independent test set PDB2272. It is evident that our proposed method can be a tool to assist in the identification of DNA-binding proteins.