Bidirectional Encoder Representations From Transformers Research Articles

A deep neural network model for Chinese toponym matching with geographic pre-training model

ABSTRACT Multiple tasks within the field of geographical information retrieval and geographical information sciences necessitate toponym matching, which involves the challenge of aligning toponyms that share a common referent. The multiple string similarity approaches struggle when confronted with the complexities associated with unofficial and/or historical variants of identical toponyms. Also, current state-of-the-art approaches/tools to supervised machine learning rely on labeled samples, and they do not adequately address the intricacies of character replacements either from transliterations or historical shifts in linguistic and cultural norms. To address these issues, this paper proposes a novel matching approach that leverages a deep neural network model empowered by geographic language representation model, known as GeoBERT, which stands for geographic Bidirectional Encoder Representations from Transformers (BERT). This model harnesses the groundbreaking capabilities of the GeoBERT framework by extending a generalized Enhanced Sequential Inference Model architecture and integrating multiple features to enhance the accuracy and robustness of the toponym matching. We present a comprehensive evaluation of the proposed method’s performance using three extensive datasets. The findings clearly illustrate that our approach outperforms the individual similarity metrics used in previous studies.

Data-Driven Strategies for Complex System Forecasts: The Role of Textual Big Data and State-Space Transformers in Decision Support

In this research, an innovative state space-based Transformer model is proposed to address the challenges of complex system prediction tasks. By integrating state space theory, the model aims to enhance the capability to capture dynamic changes in complex data, thereby improving the accuracy and robustness of prediction tasks. Extensive experimental validations were conducted on three representative tasks, including legal case judgment, legal case translation, and financial data analysis to assess the performance and application potential of the model. The experimental results demonstrate significant performance improvements of the proposed model over traditional Transformer models and other advanced variants such as Bidirectional Encoder Representation from Transformers (BERT) and Finsformer across all evaluated tasks. Specifically, in the task of legal case judgment, the proposed model exhibited a precision of 0.93, a recall of 0.90, and an accuracy of 0.91, significantly surpassing the traditional Transformer model (with precision of 0.78, recall of 0.73, accuracy of 0.76) and performances of other comparative models. In the task of legal case translation, the precision of the proposed model reached 0.95, with a recall of 0.91 and an accuracy of 0.93, also outperforming other models. Likewise, in the task of financial data analysis, the proposed model also demonstrated excellent performance, with a precision of 0.94, recall of 0.90, and accuracy of 0.92. The state space-based Transformer model proposed not only theoretically expands the research boundaries of deep learning models in complex system prediction but also validates its efficiency and broad application prospects through experiments. These achievements provide new insights and directions for future research and development of deep learning models, especially in tasks requiring the understanding and prediction of complex system dynamics.

Hype or hope? Ketamine for the treatment of depression: results from the application of deep learning to Twitter posts from 2010 to 2023

ObjectiveTo investigate societal perceptions of ketamine’s use in depression therapy by analysing Twitter posts from January 1, 2010 to April 1, 2023.MethodsUsing Twitter as the social media platform of choice, and employing search terms based on (depression OR depressed OR depressive) AND (ketamine OR esketamine OR Spravato), we collected English-language tweets from January 1, 2010, to April 1, 2023. Using unsupervised machine learning and natural language processing (NLP) techniques, including Bidirectional Encoder Representations from Transformers (BERT) and BERTopic, the study identified prevalent topics surrounding public chatter around the use of ketamine in depression treatment. Manual thematic analyses further refined these topics into themes.ResultsOut of an initial dataset of 99,405 tweets, after removing duplicate tweets, re-tweets and tweets posted by organizations over Twitter, 18,899 unique tweets from presumably individual users were analysed. Analysis of temporal trends revealed a shift in public attitudes, particularly after the United States Food and Drug Administration (FDA)’s 2019 approval of ketamine for depression. Three major themes emerged: a changing regulatory landscape, cautious optimism, and personal experiences with the drug. There was an initial spike in discussions post-FDA approval in 2019. Thereafter, cautious optimism (Theme 2) decreased among the general public, with more personal accounts (Theme 3) highlighting the potential benefits for some treatment-resistant patients. Limitations of the study include Twitter’s inherent biases towards younger, English-speaking demographics.ConclusionIn summary, the public’s multifaceted perception leans towards a hopeful stance on ketamine’s therapeutic potential for depression.

Differing Content and Language Based on Poster-Patient Relationships on the Chinese Social Media Platform Weibo: Text Classification, Sentiment Analysis, and Topic Modeling of Posts on Breast Cancer

Background Breast cancer affects the lives of not only those diagnosed but also the people around them. Many of those affected share their experiences on social media. However, these narratives may differ according to who the poster is and what their relationship with the patient is; a patient posting about their experiences may post different content from someone whose friends or family has breast cancer. Weibo is 1 of the most popular social media platforms in China, and breast cancer–related posts are frequently found there. Objective With the goal of understanding the different experiences of those affected by breast cancer in China, we aimed to explore how content and language used in relevant posts differ according to who the poster is and what their relationship with the patient is and whether there are differences in emotional expression and topic content if the patient is the poster themselves or a friend, family member, relative, or acquaintance. Methods We used Weibo as a resource to examine how posts differ according to the different poster-patient relationships. We collected a total of 10,322 relevant Weibo posts. Using a 2-step analysis method, we fine-tuned 2 Chinese Robustly Optimized Bidirectional Encoder Representations from Transformers (BERT) Pretraining Approach models on this data set with annotated poster-patient relationships. These models were lined in sequence, first a binary classifier (no_patient or patient) and then a multiclass classifier (post_user, family_members, friends_relatives, acquaintances, heard_relation), to classify poster-patient relationships. Next, we used the Linguistic Inquiry and Word Count lexicon to conduct sentiment analysis from 5 emotion categories (positive and negative emotions, anger, sadness, and anxiety), followed by topic modeling (BERTopic). Results Our binary model (F1-score=0.92) and multiclass model (F1-score=0.83) were largely able to classify poster-patient relationships accurately. Subsequent sentiment analysis showed significant differences in emotion categories across all poster-patient relationships. Notably, negative emotions and anger were higher for the “no_patient” class, but sadness and anxiety were higher for the “family_members” class. Focusing on the top 30 topics, we also noted that topics on fears and anger toward cancer were higher in the “no_patient” class, but topics on cancer treatment were higher in the “family_members” class. Conclusions Chinese users post different types of content, depending on the poster- poster-patient relationships. If the patient is family, posts are sadder and more anxious but also contain more content on treatments. However, if no patient is detected, posts show higher levels of anger. We think that these may stem from rants from posters, which may help with emotion regulation and gathering social support.

What do you think caused your ALS? An analysis of the CDC national amyotrophic lateral sclerosis patient registry qualitative risk factor data using artificial intelligence and qualitative methodology

Objective: Amyotrophic lateral sclerosis (ALS) is an incurable, progressive neurodegenerative disease with a significant health burden and poorly understood etiology. This analysis assessed the narrative responses from 3,061 participants in the Centers for Disease Control and Prevention’s National ALS Registry who answered the question, “What do you think caused your ALS?” Methods: Data analysis used qualitative methods and artificial intelligence (AI) using natural language processing (NLP), specifically, Bidirectional Encoder Representations from Transformers (BERT) to explore responses regarding participants’ perceptions of the cause of their disease. Results: Both qualitative and AI analysis methods revealed several, often aligned themes, which pointed to perceived causes including genetic, environmental, and military exposures. However, the qualitative analysis revealed detailed themes and subthemes, providing a more comprehensive understanding of participants’ perceptions. Although there were areas of alignment between AI and qualitative analysis, AI’s broader categories did not capture the nuances discovered using the more traditional, qualitative approach. The qualitative analysis also revealed that the potential causes of ALS were described within narratives that sometimes indicate self-blame and other maladaptive coping mechanisms. Conclusions: This analysis highlights the diverse range of factors that individuals with ALS consider as perceived causes for their disease. Understanding these perceptions can help clinicians to better support people living with ALS (PLWALS). The analysis highlights the benefits of using traditional qualitative methods to supplement or improve upon AI-based approaches. This rapidly evolving area of data science has the potential to remove barriers to accessing the rich narratives of people with lived experience.

Multi-class multi-label classification of social media texts for typhoon damage assessment: a two-stage model fully integrating the outputs of the hidden layers of BERT

ABSTRACT With the development of social media, it has become increasingly important to quickly and accurately identify social media texts related to disasters (e.g. typhoon) to aid in rescue and recovery efforts. Currently, multi-class classification and pre-trained language model Bidirectional Encoder Representations from Transformers (BERT) are widely used for text classification. However, most studies on typhoon damage classification are multi-class single-label, which contradicts to the reality that a social media text may correspond to multiple types of damage. Moreover, the outputs of the hidden layers of BERT are not fully utilized. This paper proposes a two-stage multi-class multi-label classification method for typhoon damage assessment by fully integrating the outputs of the hidden layers of BERT. In the first stage, sentence vectors are adopted to identify typhoon damage-related texts. In the second stage, word matrices are applied for multi-class multi-label classification to further classify the texts into five damage categories (i.e. transportation, public, electricity, forestry, and waterlogging). The two stages are trained end-to-end to identify typhoon damage from social media texts. Experiments on SinaWeibo texts during typhoon landfall in Chinese coastal regions demonstrate that the proposed method can effectively improve the accuracy of text classification and comprehensively assess typhoon damage.

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

BackgroundPretraining large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing. With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from free text has improved significantly in both the general and medical domains. However, it is difficult to train specific BERT models to perform well in domains for which few databases of a high quality and large size are publicly available. ObjectiveWe hypothesized that this problem could be addressed by oversampling a domain-specific corpus and using it for pretraining with a larger corpus in a balanced manner. In the present study, we verified our hypothesis by developing pretraining models using our method and evaluating their performance. MethodsOur proposed method was based on the simultaneous pretraining of models with knowledge from distinct domains after oversampling. We conducted three experiments in which we generated (1) English biomedical BERT from a small biomedical corpus, (2) Japanese medical BERT from a small medical corpus, and (3) enhanced biomedical BERT pretrained with complete PubMed abstracts in a balanced manner. We then compared their performance with those of conventional models. ResultsOur English BERT pretrained using both general and small medical domain corpora performed sufficiently well for practical use on the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our proposed method was more effective than the conventional methods for each biomedical corpus of the same corpus size in the general domain. Our Japanese medical BERT outperformed the other BERT models built using a conventional method for almost all the medical tasks. The model demonstrated the same trend as that of the first experiment in English. Further, our enhanced biomedical BERT model, which was not pretrained on clinical notes, achieved superior clinical and biomedical scores on the BLUE benchmark with an increase of 0.3 points in the clinical score and 0.5 points in the biomedical score. These scores were above those of the models trained without our proposed method. ConclusionsWell-balanced pretraining using oversampling instances derived from a corpus appropriate for the target task allowed us to construct a high-performance BERT model.

Deteksi Cyberbullying Menggunakan BERT dan Bi-LSTM

Cyberbullying is a digital problem that is not a new phenomenon. This existed before the advent of social networks, and cyberbullying has a wide impact, including a person's mental and physiological conditions such as sadness, anxiety and depression. The main objective of this research is to develop an effective cyberbullying detection system using natural language processing techniques. The method used in this research includes the application of the BERT (Bi-Directional Encoder Representations from Transformers) and Bi-LSTM (Bi-Directional Long Short-Term Memory) models as a deep learning approach to analyze text and detect cyberbullying behavior. This approach allows the system to understand complex language contexts and capture patterns that traditional methods may find difficult to identify. Testing was carried out using a dataset that included various types of Indonesian language texts containing cyber bullying acts. The research results show that the combination of BERT and Bi-LSTM is able to provide superior detection performance with a high accuracy rate of 90% and the ability to identify variations of cyber bullying. This research makes a significant contribution to efforts to protect individuals from the negative impacts of cyber bullying through the development of a sophisticated and adaptive detection system.

USE OF NATURAL LANGUAGE PROCESSING TECHNIQUES TO PREDICT PATIENT SELECTION FOR TOTAL HIP ARTHROPLASTY: RESULTS FROM THE AI TO REVOLUTIONISE THE PATIENT CARE PATHWAY IN HIP AND KNEE ARTHROPLASTY (ARCHERY) PROJECT

To examine whether Natural Language Processing (NLP) using a state-of-the-art clinically based Large Language Model (LLM) could predict patient selection for Total Hip Arthroplasty (THA), across a range of routinely available clinical text sources.Data pre-processing and analyses were conducted according to the Ai to Revolutionise the patient Care pathway in Hip and Knee arthroplasty (ARCHERY) project protocol (https://www.researchprotocols.org/2022/5/e37092/). Three types of deidentified Scottish regional clinical free text data were assessed: Referral letters, radiology reports and clinic letters. NLP algorithms were based on the GatorTron model, a Bidirectional Encoder Representations from Transformers (BERT) based LLM trained on 82 billion words of de-identified clinical text. Three specific inference tasks were performed: assessment of the base GatorTron model, assessment after model-fine tuning, and external validation.There were 3911, 1621 and 1503 patient text documents included from the sources of referral letters, radiology reports and clinic letters respectively. All letter sources displayed significant class imbalance, with only 15.8%, 24.9%, and 5.9% of patients linked to the respective text source documentation having undergone surgery. Untrained model performance was poor, with F1 scores (harmonic mean of precision and recall) of 0.02, 0.38 and 0.09 respectively. This did however improve with model training, with mean scores (range) of 0.39 (0.31–0.47), 0.57 (0.48–0.63) and 0.32 (0.28–0.39) across the 5 folds of cross-validation. Performance deteriorated on external validation across all three groups but remained highest for the radiology report cohort.Even with further training on a large cohort of routinely collected free-text data a clinical LLM fails to adequately perform clinical inference in NLP tasks regarding identification of those selected to undergo THA. This likely relates to the complexity and heterogeneity of free-text information and the way that patients are determined to be surgical candidates.

Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline

Building clinical registries is an important step in clinical research and improvement of patient care quality. Natural Language Processing (NLP) methods have shown promising results in extracting valuable information from unstructured clinical notes. However, the structure and nature of clinical notes are very different from regular text that state-of-the-art NLP models are trained and tested on, and they have their own set of challenges. In this study, we propose Sentence Extractor with Keywords (SE-K), an efficient and interpretable classification approach for extracting information from clinical notes and show that it outperforms more computationally expensive methods in text classification. Following the Institutional Review Board (IRB) approval, we used SE-K and two embedding based NLP approaches (Sentence Extractor with Embeddings (SE-E) and Bidirectional Encoder Representations from Transformers (BERT)) to develop comprehensive registry of anterior cruciate ligament surgeries from 20 years of unstructured clinical data at a multi-site tertiary-care regional children's hospital. The low-resource approach (SE-K) had better performance (average AUROC of 0.94 ± 0.04) than the embedding-based approaches (SE-E: 0.93 ± 0.04 and BERT: 0.87 ± 0.09) for out of sample validation, in addition to minimum performance drop between test and out-of-sample validation. Moreover, the SE-K approach was at least six times faster (on CPU) than SE-E (on CPU) and BERT (on GPU) and provides interpretability. Our proposed approach, SE-K, can be effectively used to extract relevant variables from clinic notes to build large-scale registries, with consistently better performance compared to the more resource-intensive approaches (e.g., BERT). Such approaches can facilitate information extraction from unstructured notes for registry building, quality improvement and adverse event monitoring.

Chinese satellite frequency and orbit entity relation extraction method based on dynamic integrated learning

Given the scarcity of Satellite Frequency and Orbit (SFO) resources, it holds paramount importance to establish a comprehensive knowledge graph of SFO field (SFO-KG) and employ knowledge reasoning technology to automatically mine available SFO resources. An essential aspect of constructing SFO-KG is the extraction of Chinese entity relations. Unfortunately, there is currently no publicly available Chinese SFO entity Relation Extraction (RE) dataset. Moreover, publicly available SFO text data contain numerous NA (representing for “No Answer”) relation category sentences that resemble other relation sentences and pose challenges in accurate classification, resulting in low recall and precision for the NA relation category in entity RE. Consequently, this issue adversely affects both the accuracy of constructing the knowledge graph and the efficiency of RE processes. To address these challenges, this paper proposes a method for extracting Chinese SFO text entity relations based on dynamic integrated learning. This method includes the construction of a manually annotated Chinese SFO entity RE dataset and a classifier combining features of SFO resource data. The proposed approach combines integrated learning and pre-training models, specifically utilizing Bidirectional Encoder Representation from Transformers (BERT). In addition, it incorporates one-class classification, attention mechanisms, and dynamic feedback mechanisms to improve the performance of the RE model. Experimental results show that the proposed method outperforms the traditional methods in terms of F1 value when extracting entity relations from both balanced and long-tailed datasets.

Named entity recognition model based on Multi‐BiLSTM and competition mechanism

AbstractA named entity recognition (NER) model based on multiple bidirectional long short‐term memory networks (Multi‐BiLSTM) and competition mechanism (CM) is proposed. This model includes three parts: word vectorization module, learning module and application module. In the word vectorization module, the word vectors containing semantic features are generated based on BERT (Bidirectional Encoder Representations from Transformers). In the learning module, the multiple small‐scale BiLSTM are used to extract the contextual features of the word vectors synchronously. In the application module, the competition mechanism is adopted to select the relatively better models obtained in the learning module for recognizing entities. The results show the proposed model can reduce the parameters in the learning process, and the NER performance of the proposed model is better than that of the BERT‐BiLSTM‐CRF model. For example, the F1 can be increased by 1.02% and 0.39% for CoNLL2003 and Chinese resume datasets respectively.

BERT - Based Detection and Classification of Algorithmically Generated Domains in Malware Communication

Malware threats. Particularly ransomware, often employ Algorithmically Generated Domains (AGD) for communication with Command & Control (C&C) servers. This paper introduces a novel approach for AGD detection using the Bidirectional Encoder Representations from Transformer (BERT) model, eliminating the need for intricate feature selection or hyperparameter tuning. The proposed method effectively addresses the challenge posed by sophisticated domain generation techniques, including dictionary-based and random character approaches. Experimental results demonstrate the superior performance of the BERT model in both AGD detection and classification tasks, achieving a precision, recall, and accuracy score of 0.99. The approach proves effective against diverse domain generation algorithms, enhancing the current state-of-the-art methods for securing networks against evolving malware threats.

Business text classification with imbalanced data and moderately large label spaces for digital transformation

Digital transformation refers to an organization’s use of digital technology to improve its products, services, and operations, aligning them with evolving business requirements. To demonstrate this transformative process, we present a real-life case study where a company seeks to automate the classification of their textual data rather than relying on manual methods. Transitioning to automated classification involves deploying machine learning models, which rely on pre-labeled datasets for training and making predictions on new data. However, upon receiving the dataset from the company, we faced challenges due to the imbalanced distribution of labels and moderately large label spaces. To tackle text classification with such a business dataset, we evaluated four distinct methods for multi-label text classification: fine-tuned Bidirectional Encoder Representations from Transformers (BERT), Binary Relevance, Classifier Chains, and Label Powerset. The results revealed that fine-tuned BERT significantly outperformed the other methods across key metrics like Accuracy, F1-score, Precision, and Recall. Binary Relevance also displayed competence in handling the dataset effectively, while Classifier Chains and Label Powerset exhibited comparatively less impressive performance. These findings highlight the remarkable effectiveness of fine-tuned BERT model and the Binary Relevance classifier in multi-label text classification tasks, particularly when dealing with imbalanced training datasets and moderately large label spaces. This positions them as valuable assets for businesses aiming to automate data classification in the digital transformation era.

MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Classification Tasks

This paper presents a benchmark study of MIDI-domain music classification using the mask language modeling approach of the Bidirectional Encoder Representations from Transformers (BERT). Specifically, with five public-domain datasets of single-track polyphonic piano MIDI files, we pre-train a 12-layer Transformer model using the BERT approach and fine-tune it for four downstream classification tasks. These include two note-level classification tasks---melody extraction and velocity prediction, and two sequence-level classification tasks---artist classification and emotion classification. Compared to strong recurrent neural network (RNN)-based baselines, the BERT approach does lead to higher classification accuracy, with less than 10 epochs of fine-tuning. Ablation studies further show that the pre-training remains effective even if the MIDI data of the downstream tasks are not seen during the pre-training stage, and that freezing the self-attention layers at the fine-tuning stage only slightly degrades the performance. For reproducibility, we share the weights of our pre-trained and fine-tuned models.

Analyse the Enhancement of Sentiment Analysis in Arabic by doing a Comparative Study of Several Machine Learning Techniques

Abstract: Sentiment analysis is a crucial component of natural language processing that seeks to determine the emotional sentiment expressed in a given text. This study investigates sentiment analysis in the Arabic language through a comprehensive approach that integrates traditional machine learning methods with sophisticated deep learning models. We examine the efficacy of conventional algorithms such as Support Vector Machines (SVM) and Naive Bayes, as well as sophisticated neural network architectures such as Convolutional Neural Networks (CNNs), Long Short-Term Memory networks (LSTMs), and the Arabic variant of Bidirectional Encoder Representations from Transformers (BERT). The primary novelty of our approach is in the ensemble method, which combines many approaches to enhance the precision of sentiment categorization in Arabic text. To address the particular challenges presented by Arabic sentiment analysis, such as the intricate structure of the language and the diverse regional variations, we utilize a tailored preprocessing pipeline to effectively handle the nuances of Arabic text. Our comprehensive analysis of various datasets demonstrates that the ensemble technique outperforms individual model benchmarks and offers novel insights into the interplay between different machine learning paradigms in Arabic NLP. The results emphasize the ability of hybrid approaches to improve Arabic sentiment analysis, providing a solid basis for future research and practical applications in understanding the sentiments of Arabic consumers. This study is a significant addition to the expanding domain of Arabic Natural Language Processing (NLP). This resource offers a comprehensive and advanced methodology for utilizing machine learning and deep learning methods to comprehend and analyse the intricate aspects of sentiment in the Arabic language

Promoting knowledge recommendation in innovative engineering design: a BERT-GAT-based patent representation learning approach

Since innovation in complex product design hinges on thorough engineering knowledge application, high-quality patent recommendations foster innovation in engineering design. However, many patent knowledge recommendation studies perform patent analysis without comprehensive exploration and proper organisation of knowledge, causing a superficial understanding of patents and returning arbitrary results. To mitigate this issue, a deep learning-based approach for patent representation learning and knowledge recommendation is proposed. First, a four-dimensional patent knowledge model is defined to formalise the patent attributes that critically affect the engineering design outcomes, namely patents’ domain(D), function(F), technology(T) and citation(C). Second, to exploit patent knowledge from their content and citation relationships, a representation learning approach integrating Bidirectional Encoder Representations from Transformers(BERT) and Graph Attention Network(GAT) is introduced. Thereafter a patent knowledge space is established in which each patent is characterised by the function, technology, and citation embeddings. Third, a knowledge requirement space is also constructed by vectorising a designer’s search query via BERT model and linking it to a requirement-representing patent based on similarity. Finally, a recommender prototype is developed and showcased by the knowledge recommendation in sealing structure design tasks. Comparative experiments and application cases validate the effectiveness of our method in patent representation learning and knowledge recommendation.

Fake News Detection

Abstract: Fake News is a Global problem and is constantly increasing. There are too many people that goes through a fake news and constantly making a new and fake perspective over a subject. To develop a sustainable AI solution is the market demand for many news websites and social platform. The most the work in this era are developed on Backdated Neural Network Classifiers. To engage with latest Model over a highly rated dataset like ISOT Dataset can be a sustainable solution for Fake News Detection. BERT (Bidirectional Encoder Representations from Transformers) can dynamically calculate every input and output element and their weightage. Combining this technology with the existing ones will help to build a sustainable product with a very novel cause to eradicate and control fake news over internet. It will help to detect, eliminate and try to prevent as much as possible threats causing by Fake News

Sentiment Analysis of Modern Chinese Literature Based on Deep Learning

Recent advancements in deep learning have facilitated sentiment analysis of modern Chinese literature. By leveraging techniques such as recurrent neural networks (RNNs) and transformer models like BERT, researchers can effectively gauge the sentiment expressed within literary texts. These models learn intricate patterns and context-specific nuances, enabling them to discern the emotional tone of Chinese literature accurately.Sentiment analysis, a crucial task in natural language processing, plays a pivotal role in understanding human emotions and opinions expressed in textual data. In this paper, we propose a novel deep learning framework, termed BERT-LLSTM-DL, for sentiment analysis in Chinese literature. The framework integrates Bidirectional Encoder Representations from Transformers (BERT) for language representation, Long Short-Term Memory (LSTM) networks for sequential learning, and deep learning techniques for feature extraction. We evaluate the proposed model on a dataset comprising Chinese literature texts and achieve promising results in terms of accuracy, precision, recall, and F1-score.

Pre-Training Clustering Models to Summarize Vietnamese Texts

Our investigation aims at pre-training clustering models to summarize Vietnamese texts. For this purpose, we create a large-scale dataset by collecting Vietnamese articles from newspaper websites and extracting the plain text to build the dataset, including 1,101,101 documents. We propose a new single-document extractive text summarization model based on clustering models. Our proposal clusters the documents with the hard clustering k-means algorithm and the soft clustering LDA (Latent Dirichlet Allocation) algorithm. Then, based on the pre-training clustering models, a summary model is used to select the salient sentence in the input text to construct the summary. The empirical results showed that our summary model achieved 51.22% ROUGE-1, 17.62% ROUGE-2 and 29.16% ROUGE-L on the testing set. Besides the traditional word representation such as BoW (Bag-of-Words), we also use the word meaning-based tools like FastText and BERT (Bidirectional Encoder Representations from Transformers) in our model. The additional benefit of our proposed extractive summary model is that the output summary is a long-text, readable document. Furthermore, the model’s architecture is straightforward, easy to understand and runs on cost-efficient resources like arm CPU and GPU too.