Bidirectional Encoder Representations From Transformers Research Articles

Determining community happiness index with transformers and attention-based deep learning

In the current digital era, evaluating the quality of people's lives and their happiness index is closely related to their expressions and opinions on Twitter social media. Measuring population welfare goes beyond monetary aspects, focusing more on subjective well-being, and sentiment analysis helps evaluate people's perceptions of happiness aspects. Aspect-based sentiment analysis (ABSA) effectively identifies sentiments on predetermined aspects. The previous study has used Word-to-Vector (Word2Vec) and long short-term memory (LSTM) methods with or without attention mechanism (AM) to solve ABSA cases. However, the problem with the previous study is that Word2Vec has the disadvantage of being unable to handle the context of words in a sentence. Therefore, this study will address the problem with bidirectional encoder representations from transformers (BERT), which has the advantage of performing bidirectional training. Bayesian optimization as a hyperparameter tuning technique is used to find the best combination of parameters during the training process. Here we show that BERT-LSTM-AM outperforms the Word2Vec-LSTM-AM model in predicting aspect and sentiment. Furthermore, we found that BERT is the best state-of-the-art embedding technique for representing words in a sentence. Our results demonstrate how BERT as an embedding technique can significantly improve the model performance over Word2Vec.

Aspect based sentiment analysis using fine-tuned BERT model with deep context features

<p>Sentiment analysis is the task of analysing, processing, inferencing and concluding the subjective texts along with sentiment. Considering the application of sentiment analysis, it is categorized into document-level, sentence-level and aspect level. In past, several researches have achieved solutions through the bidirectional encoder representations from transformers (BERT) model, however, the existing model does not understand the context of the aspect in deep, which leads to low metrics. This research work leads to the study of the aspect-based sentiment analysis presented by deep context bidirectional encoder representations from transformers (DC-BERT), main aim of the DC-BERT model is to improvise the context understating for aspects to enhance the metrics. DC-BERT model comprises fine-tuned BERT model along with a deep context features layer, which enables the model to understand the context of targeted aspects deeply. A customized feature layer is introduced to extract two distinctive features, later both features are integrated through the interaction layer. DC-BERT mode is evaluated considering the review dataset of laptops and restaurants from SemEval 2014 task 4, evaluation is carried out considering the different metrics. In comparison with the other model, DC-BERT achieves an accuracy of 84.48% and 92.86% for laptop and restaurant datasets respectively.</p>

Sentiment analysis of student feedback using attention-based RNN and transformer embedding

Sentiment analysis systems aim to assess people’s opinions across various domains by collecting and categorizing feedback and reviews. In our study, researchers put forward a sentiment analysis system that leverages three distinct embedding techniques: automatic, global vectors (GloVe) for word representation, and bidirectional encoder representations from transformers (BERT). This system features an attention layer, with the best model chosen through rigorous comparisons. In developing the sentiment analysis model, we employed a hybrid dataset comprising students’ feedback and comments. This dataset comprises 3,820 comments, including 2,773 from formal evaluations and 1,047 generated by ChatGPT and prompting engineering. Our main motivation for integrating generative AI was to balance both positive and negative comments. We also explored recurrent neural network (RNN), gated recurrent unit (GRU), long short-term memory (LSTM), and bidirectional long short-term memory (Bi-LSTM), with and without pre-trained GloVe embedding. These techniques produced F-scores ranging from 67% to 69%. On the other hand, the sentiment model based on BERT, particularly its KERAS implementation, achieved higher F-scores ranging from 83% to 87%. The Bi-LSTM architecture outperformed other models and the inclusion of an attention layer further enhanced the performance, resulting in F-scores of 89% and 88% from the Bi-LSTM-BERT sentiment models, respectively.

Hybrid model for extractive single document summarization: utilizing BERTopic and BERT model

Extractive text summarization has been a popular research area for many years. The goal of this task is to generate a compact and coherent summary of a given document, preserving the most important information. However, current extractive summarization methods still face several challenges such as semantic drift, repetition, redundancy, and lack of coherence. A novel approach is presented in this paper to improve the performance of an extractive summarization model based on bidirectional encoder representations from transformers (BERT) by incorporating topic modeling using the BERTopic model. Our method first utilizes BERTopic to identify the dominant topics in a document and then employs a BERT-based deep neural network to extract the most salient sentences related to those topics. Our experiments on the cable news network (CNN)/daily mail dataset demonstrate that our proposed method outperforms state-of-the-art BERT-based extractive summarization models in terms of recall-oriented understudy for gisting evaluation (ROUGE) scores, which resulted in an increase of 32.53% of ROUGE-1, 47.55% of ROUGE-2, and 16.63% of ROUGE-L when compared to baseline BERT-based extractive summarization models. This paper contributes to the field of extractive text summarization, highlights the potential of topic modeling in improving summarization results, and provides a new direction for future research.

Aspect based sentiment analysis using a novel ensemble deep network

Aspect-based sentiment analysis (ABSA) is a fine-grained task in natural language processing, which aims to predict the sentiment polarity of several parts of a sentence or document. The essential aspect of sentiment polarity and global context have deep relationships that have not received enough attention. This research work design and develops a novel ensemble deep network (EDN) which comprises the various network and integrated to enhance the model performance. In the proposed work the words of the input sentence are converted into word vectors using the optimised bidirectional encoder representations from transformers (BERT) model and an optimised BERT-graph neural networks (GNN) model with convolutions is built that analyses the ABSA of the input sentence. The optimised GNN model with convolutions for context-based word representations is developed for the word-vector embedding. We propose a novel EDN for an ABSA model for optimised BERT over GNN with convolutions. The proposed ensemble deep network proposed system (EDN-PS) is evaluated with various existing techniques and results are plotted in terms of metrics for accuracy and F1- score, concluding that the proposed EDN-PS ensures better performance in comparison with the existing model.

Multi-step Transfer Learning in Natural Language Processing for the Health Domain

The restricted access to data in healthcare facilities due to patient privacy and confidentiality policies has led to the application of general natural language processing (NLP) techniques advancing relatively slowly in the health domain. Additionally, because clinical data is unique to various institutions and laboratories, there are not enough standards and conventions for data annotation. In places without robust death registration systems, the cause of death (COD) is determined through a verbal autopsy (VA) report. A non-clinician field agent completes a VA report using a set of standardized questions as guide to identify the symptoms of a COD. The narrative text of the VA report is used as a case study to examine the difficulties of applying NLP techniques to the healthcare domain. This paper presents a framework that leverages knowledge across multiple domains via two domain adaptation techniques: feature extraction and fine-tuning. These techniques aim to improve VA text representations for COD classification tasks in the health domain. The framework is motivated by multi-step learning, where a final learning task is realized via a sequence of intermediate learning tasks. The framework builds upon the strengths of the Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Models (ELMo) models pretrained on the general English and biomedical domains. These models are employed to extract features from the VA narratives. Our results demonstrate improved performance when initializing the learning of BERT embeddings with ELMo embeddings. The benefit of incorporating character-level information for learning word embeddings in the English domain, coupled with word-level information for learning word embeddings in the biomedical domain, is also evident.

Extracting Pulmonary Nodules and Nodule Characteristics from Radiology Reports of Lung Cancer Screening Patients Using Transformer Models

Abstract Pulmonary nodules and nodule characteristics are important indicators of lung nodule malignancy. However, nodule information is often documented as free text in clinical narratives such as radiology reports in electronic health record systems. Natural language processing (NLP) is the key technology to extract and standardize patient information from radiology reports into structured data elements. This study aimed to develop an NLP system using state-of-the-art transformer models to extract pulmonary nodules and associated nodule characteristics from radiology reports. We identified a cohort of 3080 patients who underwent LDCT at the University of Florida health system and collected their radiology reports. We manually annotated 394 reports as the gold standard. We explored eight pretrained transformer models from three transformer architectures including bidirectional encoder representations from transformers (BERT), robustly optimized BERT approach (RoBERTa), and A Lite BERT (ALBERT), for clinical concept extraction, relation identification, and negation detection. We examined general transformer models pretrained using general English corpora, transformer models fine-tuned using a clinical corpus, and a large clinical transformer model, GatorTron, which was trained from scratch using 90 billion words of clinical text. We compared transformer models with two baseline models including a recurrent neural network implemented using bidirectional long short-term memory with a conditional random fields layer and support vector machines. RoBERTa-mimic achieved the best F1-score of 0.9279 for nodule concept and nodule characteristics extraction. ALBERT-base and GatorTron achieved the best F1-score of 0.9737 in linking nodule characteristics to pulmonary nodules. Seven out of eight transformers achieved the best F1-score of 1.0000 for negation detection. Our end-to-end system achieved an overall F1-score of 0.8869. This study demonstrated the advantage of state-of-the-art transformer models for pulmonary nodule information extraction from radiology reports.

Real-Time Sentiment Analysis System Using the BERT Model

Opinion mining or we can say the Sentimental analysis is a critical issue where a large amount of information related to public opinion is widely spread in the world. So, in this paper, we will try to understand the public opinion and decision-making process in real time for the given text data. This proposed system aims to develop a real-time sentiment analysis of text data from various sources, such as social media, news websites, etc. The system will involve several processes like data collection, data preprocessing, sentiment analysis using the BERT (Bidirectional Encoder Representation from Transformers) model, and real-time processing to find the sentiment trends and patterns. based on [4] we can say that the performance of the BERT model is highly accurate as compared to the other traditional models like Random Forest, KNN, and Decision Tree, etc. The result which is obtained using the BERT model shows the accuracy of the system.

Microblog sentiment analysis method using BTCBMA model in Spark big data environment

Abstract Microblogs are currently one of the most well-liked social platforms in China, and sentiment analysis of microblog texts can help further analyze the realization of their media value; however, the current task of sentiment analysis based on microblog information suffers from low accuracy due to the large size and high redundancy of microblog data, a microblog sentiment analysis method using Bidirectional Encoder Representation from Transformers (BERT)–Text Convolutional Neural Network (TextCNN)–Bidirectional Gate Recurrent Unit (BiGRU)–Multihead-Attention model in Spark big data environment is proposed. First, the Chinese pre-trained language model BERT is used to convert the input data into dynamic character-level word vectors; then, TextCNN is used to effectively obtain local features such as keywords and pool the filtered features; then, BiGRU is introduced to quickly capture more comprehensive semantic information; finally, a multi-headed attention mechanism is implemented to emphasize the most significant features in order to accomplish the sentiment classification of microblog information task precisely. By comparing the existing advanced models, the proposed model demonstrates an improvement of at least 4.99% and 0.05 in accuracy and F1-score evaluation indexes, respectively. This enhancement significantly enhances the accuracy of microblog sentiment analysis tasks and aids pertinent authorities in comprehending the inclination of individual’s attitude toward hot topics. Furthermore, it facilitates a prompt prediction of topic trends, enabling them to guide public opinion accordingly.

A deep neural network model for Chinese toponym matching with geographic pre-training model

ABSTRACT Multiple tasks within the field of geographical information retrieval and geographical information sciences necessitate toponym matching, which involves the challenge of aligning toponyms that share a common referent. The multiple string similarity approaches struggle when confronted with the complexities associated with unofficial and/or historical variants of identical toponyms. Also, current state-of-the-art approaches/tools to supervised machine learning rely on labeled samples, and they do not adequately address the intricacies of character replacements either from transliterations or historical shifts in linguistic and cultural norms. To address these issues, this paper proposes a novel matching approach that leverages a deep neural network model empowered by geographic language representation model, known as GeoBERT, which stands for geographic Bidirectional Encoder Representations from Transformers (BERT). This model harnesses the groundbreaking capabilities of the GeoBERT framework by extending a generalized Enhanced Sequential Inference Model architecture and integrating multiple features to enhance the accuracy and robustness of the toponym matching. We present a comprehensive evaluation of the proposed method’s performance using three extensive datasets. The findings clearly illustrate that our approach outperforms the individual similarity metrics used in previous studies.

Data-Driven Strategies for Complex System Forecasts: The Role of Textual Big Data and State-Space Transformers in Decision Support

In this research, an innovative state space-based Transformer model is proposed to address the challenges of complex system prediction tasks. By integrating state space theory, the model aims to enhance the capability to capture dynamic changes in complex data, thereby improving the accuracy and robustness of prediction tasks. Extensive experimental validations were conducted on three representative tasks, including legal case judgment, legal case translation, and financial data analysis to assess the performance and application potential of the model. The experimental results demonstrate significant performance improvements of the proposed model over traditional Transformer models and other advanced variants such as Bidirectional Encoder Representation from Transformers (BERT) and Finsformer across all evaluated tasks. Specifically, in the task of legal case judgment, the proposed model exhibited a precision of 0.93, a recall of 0.90, and an accuracy of 0.91, significantly surpassing the traditional Transformer model (with precision of 0.78, recall of 0.73, accuracy of 0.76) and performances of other comparative models. In the task of legal case translation, the precision of the proposed model reached 0.95, with a recall of 0.91 and an accuracy of 0.93, also outperforming other models. Likewise, in the task of financial data analysis, the proposed model also demonstrated excellent performance, with a precision of 0.94, recall of 0.90, and accuracy of 0.92. The state space-based Transformer model proposed not only theoretically expands the research boundaries of deep learning models in complex system prediction but also validates its efficiency and broad application prospects through experiments. These achievements provide new insights and directions for future research and development of deep learning models, especially in tasks requiring the understanding and prediction of complex system dynamics.

What do you think caused your ALS? An analysis of the CDC national amyotrophic lateral sclerosis patient registry qualitative risk factor data using artificial intelligence and qualitative methodology

Objective: Amyotrophic lateral sclerosis (ALS) is an incurable, progressive neurodegenerative disease with a significant health burden and poorly understood etiology. This analysis assessed the narrative responses from 3,061 participants in the Centers for Disease Control and Prevention’s National ALS Registry who answered the question, “What do you think caused your ALS?” Methods: Data analysis used qualitative methods and artificial intelligence (AI) using natural language processing (NLP), specifically, Bidirectional Encoder Representations from Transformers (BERT) to explore responses regarding participants’ perceptions of the cause of their disease. Results: Both qualitative and AI analysis methods revealed several, often aligned themes, which pointed to perceived causes including genetic, environmental, and military exposures. However, the qualitative analysis revealed detailed themes and subthemes, providing a more comprehensive understanding of participants’ perceptions. Although there were areas of alignment between AI and qualitative analysis, AI’s broader categories did not capture the nuances discovered using the more traditional, qualitative approach. The qualitative analysis also revealed that the potential causes of ALS were described within narratives that sometimes indicate self-blame and other maladaptive coping mechanisms. Conclusions: This analysis highlights the diverse range of factors that individuals with ALS consider as perceived causes for their disease. Understanding these perceptions can help clinicians to better support people living with ALS (PLWALS). The analysis highlights the benefits of using traditional qualitative methods to supplement or improve upon AI-based approaches. This rapidly evolving area of data science has the potential to remove barriers to accessing the rich narratives of people with lived experience.

Multi-class multi-label classification of social media texts for typhoon damage assessment: a two-stage model fully integrating the outputs of the hidden layers of BERT

ABSTRACT With the development of social media, it has become increasingly important to quickly and accurately identify social media texts related to disasters (e.g. typhoon) to aid in rescue and recovery efforts. Currently, multi-class classification and pre-trained language model Bidirectional Encoder Representations from Transformers (BERT) are widely used for text classification. However, most studies on typhoon damage classification are multi-class single-label, which contradicts to the reality that a social media text may correspond to multiple types of damage. Moreover, the outputs of the hidden layers of BERT are not fully utilized. This paper proposes a two-stage multi-class multi-label classification method for typhoon damage assessment by fully integrating the outputs of the hidden layers of BERT. In the first stage, sentence vectors are adopted to identify typhoon damage-related texts. In the second stage, word matrices are applied for multi-class multi-label classification to further classify the texts into five damage categories (i.e. transportation, public, electricity, forestry, and waterlogging). The two stages are trained end-to-end to identify typhoon damage from social media texts. Experiments on SinaWeibo texts during typhoon landfall in Chinese coastal regions demonstrate that the proposed method can effectively improve the accuracy of text classification and comprehensively assess typhoon damage.

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

BackgroundPretraining large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing. With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from free text has improved significantly in both the general and medical domains. However, it is difficult to train specific BERT models to perform well in domains for which few databases of a high quality and large size are publicly available. ObjectiveWe hypothesized that this problem could be addressed by oversampling a domain-specific corpus and using it for pretraining with a larger corpus in a balanced manner. In the present study, we verified our hypothesis by developing pretraining models using our method and evaluating their performance. MethodsOur proposed method was based on the simultaneous pretraining of models with knowledge from distinct domains after oversampling. We conducted three experiments in which we generated (1) English biomedical BERT from a small biomedical corpus, (2) Japanese medical BERT from a small medical corpus, and (3) enhanced biomedical BERT pretrained with complete PubMed abstracts in a balanced manner. We then compared their performance with those of conventional models. ResultsOur English BERT pretrained using both general and small medical domain corpora performed sufficiently well for practical use on the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our proposed method was more effective than the conventional methods for each biomedical corpus of the same corpus size in the general domain. Our Japanese medical BERT outperformed the other BERT models built using a conventional method for almost all the medical tasks. The model demonstrated the same trend as that of the first experiment in English. Further, our enhanced biomedical BERT model, which was not pretrained on clinical notes, achieved superior clinical and biomedical scores on the BLUE benchmark with an increase of 0.3 points in the clinical score and 0.5 points in the biomedical score. These scores were above those of the models trained without our proposed method. ConclusionsWell-balanced pretraining using oversampling instances derived from a corpus appropriate for the target task allowed us to construct a high-performance BERT model.

Deteksi Cyberbullying Menggunakan BERT dan Bi-LSTM

Cyberbullying is a digital problem that is not a new phenomenon. This existed before the advent of social networks, and cyberbullying has a wide impact, including a person's mental and physiological conditions such as sadness, anxiety and depression. The main objective of this research is to develop an effective cyberbullying detection system using natural language processing techniques. The method used in this research includes the application of the BERT (Bi-Directional Encoder Representations from Transformers) and Bi-LSTM (Bi-Directional Long Short-Term Memory) models as a deep learning approach to analyze text and detect cyberbullying behavior. This approach allows the system to understand complex language contexts and capture patterns that traditional methods may find difficult to identify. Testing was carried out using a dataset that included various types of Indonesian language texts containing cyber bullying acts. The research results show that the combination of BERT and Bi-LSTM is able to provide superior detection performance with a high accuracy rate of 90% and the ability to identify variations of cyber bullying. This research makes a significant contribution to efforts to protect individuals from the negative impacts of cyber bullying through the development of a sophisticated and adaptive detection system.

USE OF NATURAL LANGUAGE PROCESSING TECHNIQUES TO PREDICT PATIENT SELECTION FOR TOTAL HIP ARTHROPLASTY: RESULTS FROM THE AI TO REVOLUTIONISE THE PATIENT CARE PATHWAY IN HIP AND KNEE ARTHROPLASTY (ARCHERY) PROJECT

To examine whether Natural Language Processing (NLP) using a state-of-the-art clinically based Large Language Model (LLM) could predict patient selection for Total Hip Arthroplasty (THA), across a range of routinely available clinical text sources.Data pre-processing and analyses were conducted according to the Ai to Revolutionise the patient Care pathway in Hip and Knee arthroplasty (ARCHERY) project protocol (https://www.researchprotocols.org/2022/5/e37092/). Three types of deidentified Scottish regional clinical free text data were assessed: Referral letters, radiology reports and clinic letters. NLP algorithms were based on the GatorTron model, a Bidirectional Encoder Representations from Transformers (BERT) based LLM trained on 82 billion words of de-identified clinical text. Three specific inference tasks were performed: assessment of the base GatorTron model, assessment after model-fine tuning, and external validation.There were 3911, 1621 and 1503 patient text documents included from the sources of referral letters, radiology reports and clinic letters respectively. All letter sources displayed significant class imbalance, with only 15.8%, 24.9%, and 5.9% of patients linked to the respective text source documentation having undergone surgery. Untrained model performance was poor, with F1 scores (harmonic mean of precision and recall) of 0.02, 0.38 and 0.09 respectively. This did however improve with model training, with mean scores (range) of 0.39 (0.31–0.47), 0.57 (0.48–0.63) and 0.32 (0.28–0.39) across the 5 folds of cross-validation. Performance deteriorated on external validation across all three groups but remained highest for the radiology report cohort.Even with further training on a large cohort of routinely collected free-text data a clinical LLM fails to adequately perform clinical inference in NLP tasks regarding identification of those selected to undergo THA. This likely relates to the complexity and heterogeneity of free-text information and the way that patients are determined to be surgical candidates.

Chinese satellite frequency and orbit entity relation extraction method based on dynamic integrated learning

Given the scarcity of Satellite Frequency and Orbit (SFO) resources, it holds paramount importance to establish a comprehensive knowledge graph of SFO field (SFO-KG) and employ knowledge reasoning technology to automatically mine available SFO resources. An essential aspect of constructing SFO-KG is the extraction of Chinese entity relations. Unfortunately, there is currently no publicly available Chinese SFO entity Relation Extraction (RE) dataset. Moreover, publicly available SFO text data contain numerous NA (representing for “No Answer”) relation category sentences that resemble other relation sentences and pose challenges in accurate classification, resulting in low recall and precision for the NA relation category in entity RE. Consequently, this issue adversely affects both the accuracy of constructing the knowledge graph and the efficiency of RE processes. To address these challenges, this paper proposes a method for extracting Chinese SFO text entity relations based on dynamic integrated learning. This method includes the construction of a manually annotated Chinese SFO entity RE dataset and a classifier combining features of SFO resource data. The proposed approach combines integrated learning and pre-training models, specifically utilizing Bidirectional Encoder Representation from Transformers (BERT). In addition, it incorporates one-class classification, attention mechanisms, and dynamic feedback mechanisms to improve the performance of the RE model. Experimental results show that the proposed method outperforms the traditional methods in terms of F1 value when extracting entity relations from both balanced and long-tailed datasets.

Named entity recognition model based on Multi‐BiLSTM and competition mechanism

AbstractA named entity recognition (NER) model based on multiple bidirectional long short‐term memory networks (Multi‐BiLSTM) and competition mechanism (CM) is proposed. This model includes three parts: word vectorization module, learning module and application module. In the word vectorization module, the word vectors containing semantic features are generated based on BERT (Bidirectional Encoder Representations from Transformers). In the learning module, the multiple small‐scale BiLSTM are used to extract the contextual features of the word vectors synchronously. In the application module, the competition mechanism is adopted to select the relatively better models obtained in the learning module for recognizing entities. The results show the proposed model can reduce the parameters in the learning process, and the NER performance of the proposed model is better than that of the BERT‐BiLSTM‐CRF model. For example, the F1 can be increased by 1.02% and 0.39% for CoNLL2003 and Chinese resume datasets respectively.

BERT - Based Detection and Classification of Algorithmically Generated Domains in Malware Communication

Malware threats. Particularly ransomware, often employ Algorithmically Generated Domains (AGD) for communication with Command & Control (C&C) servers. This paper introduces a novel approach for AGD detection using the Bidirectional Encoder Representations from Transformer (BERT) model, eliminating the need for intricate feature selection or hyperparameter tuning. The proposed method effectively addresses the challenge posed by sophisticated domain generation techniques, including dictionary-based and random character approaches. Experimental results demonstrate the superior performance of the BERT model in both AGD detection and classification tasks, achieving a precision, recall, and accuracy score of 0.99. The approach proves effective against diverse domain generation algorithms, enhancing the current state-of-the-art methods for securing networks against evolving malware threats.

Business text classification with imbalanced data and moderately large label spaces for digital transformation

Digital transformation refers to an organization’s use of digital technology to improve its products, services, and operations, aligning them with evolving business requirements. To demonstrate this transformative process, we present a real-life case study where a company seeks to automate the classification of their textual data rather than relying on manual methods. Transitioning to automated classification involves deploying machine learning models, which rely on pre-labeled datasets for training and making predictions on new data. However, upon receiving the dataset from the company, we faced challenges due to the imbalanced distribution of labels and moderately large label spaces. To tackle text classification with such a business dataset, we evaluated four distinct methods for multi-label text classification: fine-tuned Bidirectional Encoder Representations from Transformers (BERT), Binary Relevance, Classifier Chains, and Label Powerset. The results revealed that fine-tuned BERT significantly outperformed the other methods across key metrics like Accuracy, F1-score, Precision, and Recall. Binary Relevance also displayed competence in handling the dataset effectively, while Classifier Chains and Label Powerset exhibited comparatively less impressive performance. These findings highlight the remarkable effectiveness of fine-tuned BERT model and the Binary Relevance classifier in multi-label text classification tasks, particularly when dealing with imbalanced training datasets and moderately large label spaces. This positions them as valuable assets for businesses aiming to automate data classification in the digital transformation era.