Bidirectional Encoder Representations From Transformers Research Articles

A hybrid BERT-CPSO model for multi-class depression detection using pure hindi and hinglish multimodal data on social media

Due to the psychological strain that depression causes, there has been a noticeable increase in the number of persons compromising their lives in recent years. Social media platforms provide researchers with an entirely novel viewpoint on identifying individuals who are depressed. Previous research on automatic learning models for depression detection revealed low detection accuracy and an absence of optimizing techniques that could enhance detection accuracy. Furthermore, there is no such dataset, and very little study has been done on the multimodal pure Hindi and code-mixed Hinglish language domains. In light of this, we developed a Hindi dataset and suggested reliable methods for depression detection based on multimodal data, i.e., text and images, using the Hindi and Hinglish languages. This study aims to accomplish three things: first, it will evaluate text data using an effective Bidirectional Encoder Representations from Transformers (BERT) approach and compare it with other transfer learning variants; second, it will analyze image data by suggesting a Convolutional Neural Network (CNN) optimized with a nature-inspired algorithm, namely Particle Swarm Optimization (PSO), or CPSO; and third, it will classify the multimodal data into depressive and non-depressive posts by suggesting a hybrid of the best-performing models on text and images, namely BERT-CPSO (BTCPSO). The results produced with the BERT model showed the best accuracy of 95% for text data, in contrast to RoBERTa, DistilBERT, and XLNet. Further, CPSO outperforms other Machine Learning (ML) and Deep Learning (DL) algorithms for image data with an accuracy of 95%. Additionally, comparing the proposed CPSO with a basic CNN revealed that integrating the PSO technique with CNN increased the model's accuracy in detecting depressed posts by 5%. In conclusion, hybrid BERT-CPSO outperforms other BERT combinations with ML and DL algorithms for multimodal data, achieving 97%, 95%, 98%, and 96%, respectively, in accuracy, recall, precision, and F1-scores. As a result, the findings of comparing the suggested technique with the earlier models show the effectiveness of the approach that has been provided and can help medical professionals diagnose depression with precision.

Machine learning algorithms for breast cancer analysis: performance and accuracy comparison

Breast cancer, a leading cause of cancer mortality among women, necessitates early detection to improve survival rates. Traditional diagnostics face accuracy and speed limitations, prompting this study to explore machine learning for enhanced diagnostics. We applied bidirectional encoder representations from transformers (BERT), long short-term memory (LSTM), naive Bayes, support vector machine (SVM), and random forest to the Breast Cancer Wisconsin dataset, implementing a thorough methodology involving data preprocessing, feature extraction, and model validation. BERT led in accuracy at 92.5%, showcasing advanced algorithms' potential in medical diagnostics, with random forest 90.6%, SVM 89.3%, LSTM 88.7%, and naive Bayes 85.2%; also showing promising results. The study underscores the importance of incorporating machine learning, especially BERT, into clinical decision-making, potentially revolutionizing breast cancer diagnostics by improving accuracy and efficiency. We recommend healthcare practitioners integrate these algorithms into their diagnostic processes. Future research should reeefine these algorithms and extend their application to enhance patient care further.

Sentiment-electroencephalogram fusion for efficient product review prediction using correlation-based deep learning neural network

<span lang="EN-US">Various techniques have been proposed and implemented in previous work for sentiment analysis prediction. However, achieving satisfactory quality of description and fault prediction remains a challenging task. To overcome these limitations, this study proposes an efficient prediction technique that utilizes sentiment analysis of product reviews and electroencephalogram (EEG) signals using correlation-based deep learning neural network (CDNN). The study employs two types of datasets: EEG signals and Amazon product reviews. During the pre-processing phase, EEG signals undergo normalization, while Amazon product reviews undergo tokenization, stop word removal, and weighting factor application to convert unstructured data into a structured format. Subsequently, the pre-processed EEG signals and reviews are analyzed to extract features like emotion, demographic information, personality traits, and sentiment. These features are then employed in sentiment analysis via an entropy-based deep-learning neural network. The proposed CDNN utilizes the grasshopper optimization algorithm (EGOA) to optimize hyperparameters for each layer. Comparative performance assessment against established methods like convolutional neural network (CNN), <a name="_Hlk167111148"></a>long short-term memory (LSTM), multiclass <a name="_Hlk175651957"></a>support vector machine (M-SVM), and bidirectional encoder representations from transformers (BERT) is conducted, and the results are evaluated. Experimental result reveal that the proposed system outperforms traditional approaches.</span>

Customer Sentiment Analysis for E-Commerce: A Hybrid approach using CNNs & BERT

As the online shopping industry continues to grow at a staggering rate, e-commerce channels have also begun the use of automated systems for sentiment analysis so as to be able to assess comprehension of user reviews. Over the past few years, machine learning (ML) has come of age with the introduction of deep learning models such as Convolutional Neural Networks (CNNs) and Bidirectional Encoder Representations from Transformers (BERT) which have greatly improved the levels of accuracy achieved in sentiment analysis. This paper aims to address this problem by proposing a hybrid model, based on networks of convolutional neural networks (CNNs) and BERT, for the purposes of e-commerce sentiment classification. Posting more plausible situations, academic findings, and perspectives for future research, we focus more on practical issues. When we compared traditional approaches with our hybrid, it became clear that the model performs better in many dimensions: accuracy, precision, and progressively recall

Deep Learning-Based Identification of Immoral Posts on Social Media Using Fine-tuned Bert Model

The propagation of immoral content on social media poses substantial worries to online societal well-being and communication standards. While beneficial, traditional machine learning (ML) methods fall short of capturing the difficulty of textual and sequential data. This work reports this gap by suggesting a deep learning-based technique for detecting immoral posts on social media. The proposed model presents a fine-tuned Bidirectional Encoder representation from Transformers (BERT) with word embedding methods. Word2Vec and Global Vectors for Word Representation (GloVe) are employed to improve the identification of immoral posts on social media platforms to advance detection accuracy and strength. The incentive behind this study stems from the increasing demand for more sophisticated methods to struggle with damaging content. The proposed model is considered to capture the complicated patterns and semantic nuances in immoral posts by decreasing the dependence on manual feature engineering. The model is trained and assessed using benchmark datasets containing SARC and HatEval, which deliver a detailed set of labelled user-generated posts. The proposed model shows the best performance compared to traditional ML approaches. The fine-tuned Bert-based Word2Vec embeddings achieved a precision of 95.68%, recall of 96.85 %, and F1 scores of 96.26% on the SARC dataset. Fine-tuned Bert-based GloVe on the HatEval dataset achieved superior precision of 96.65, recall of 97.75, and F1-score of 97.20. The proposed results highlight the potential of the deep learning (DL) approach and fine-tuned BERT models, considerably refining the detection of unethical content on social networks.

Mitigating Bias in Large Language Models: A Multi-Task Training Approach Using BERT

Abstract. Large language models (LLMs), such as ChatGPT, have become essential tools due to their advanced natural language processing capabilities. However, these models, trained on extensive internet text, can inadvertently learn and propagate unwanted biases, impacting their outputs. This study addresses this issue by analyzing and mitigating such biases through a multi-task and multi-stage training approach. Utilizing the Winograd Bias (Winobias) dataset, the research fine-tunes the Bidirectional Encoder Representations from Transformers (BERT) model to reduce biased outputs. The approach includes an initial mask task to establish a general understanding and a subsequent cloze task to specifically target and mitigate biases. Results demonstrate a significant reduction in bias, with the original model showing approximately 90% certainty in biased outputs, whereas the de-biased model reduced this certainty to 55%. This study effectively showcases a method for bias reduction by modifying only a few parameters, emphasizing a practical approach to enhancing fairness and balance in LLMs used across various applications.

Comparative Evaluation of Sentiment Analysis Methods: From Traditional Techniques to Advanced Deep Learning Models

Abstract. Sentiment evaluation plays a crucial role in deciphering public perception and consumer responses in today's digital landscape. This investigation offers a thorough assessment of diverse sentiment evaluation techniques, contrasting conventional machine learning methodologies with cutting-edge deep learning frameworks. In particular, the research scrutinizes the efficacy of Bidirectional Encoder Representations from Transformers (BERT)-derived architectures (BERT-Base and Robustly Optimized BERT Pretraining Approach (RoBERTa)), Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM), Support Vector Machines (SVM), and Naive Bayes classifiers. The study gauges these approaches based on their precision, recall, F1-metric, overall accuracy, and computational efficiency using an extensive sentiment evaluation dataset. The results reveal that BERT-based models, particularly RoBERTa, achieve the highest accuracy (87.44%) and F1-score (0.8746), though they also require the longest training time (approximately 3 hours). CNN and LSTM models strike a balance between performance and efficiency, while traditional methods like SVM and Naive Bayes offer faster training and deployment with moderate accuracy. The insights gained from this study are valuable for both researchers and practitioners, highlighting the trade-offs between model performance, computational demands, and practical deployment considerations in sentiment analysis applications.

Exploring the Impact of Word2Vec Embeddings Across Neural Network Architectures for Sentiment Analysis

Abstract. Sentiment analysis is crucial for understanding public opinion, gauging customer satisfaction, and making informed business decisions based on the emotional tone of textual data. This study investigates the performance of different Word2Vec-based embedding strategies static, non-static, and multichannel for sentiment analysis across various neural network architectures, including Convolution Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Gated Recurrent Units (GRUs). Despite the rise of advanced contextual embedding methods such as Bidirectional Encoder Representations from Transformers (BERT), Word to Vector (Word2Vec) retains its importance due to its simplicity and lower computational demands, making it ideal for use in settings with limited resources. The goal is to evaluate the impact of fine-tuning Word2Vec embeddings on the accuracy of sentiment classification. Using the Internet Movie Database (IMDb), this work finds that multichannel embeddings, which combine static and non-static representations, provide the best performance across most architectures, while static embeddings continue to deliver strong results in specific sequential models. These findings highlight the balance between efficiency and accuracy in traditional word embeddings, particularly when advanced models are not feasible.

Clustering-based topic modeling for biomedical documents extractive text summarization

AbstractThe increasing volume of electronic text, especially in the biomedical domain, requires automatic text summarization (ATS) to help researchers navigate and find relevant information. This study proposes an unsupervised extractive ATS method to improve the quality of biomedical document summaries by focusing on subtopic diversity. The method integrates topic modeling and clustering with bidirectional encoder representation from transformers (BERT). To evaluate the effectiveness of the proposed study, it tested on a new corpus of 200 biomedical research papers from Biomed Central. The results were evaluated using the ROUGE metric and qualitative evaluation by medical experts. The ROUGE metric yielded scores of 0.4838 (Rouge-1), 0.2174 (Rouge-2), and 0.2206 (Rouge-L), while the qualitative evaluation achieved an average score of 4.10, 4.06, 3.55, 4.0, and 4.0 for completeness, relevance, conciseness, informativity, and readability, respectively. The results demonstrate the effectiveness of the proposed method in summarizing long medical documents.

Enhancing Bias Assessment for Complex Term Groups in Language Embedding Models: Quantitative Comparison of Methods.

Artificial intelligence (AI) is rapidly being adopted to build products and aid in the decision-making process across industries. However, AI systems have been shown to exhibit and even amplify biases, causing a growing concern among people worldwide. Thus, investigating methods of measuring and mitigating bias within these AI-powered tools is necessary. In natural language processing applications, the word embedding association test (WEAT) is a popular method of measuring bias in input embeddings, a common area of measure bias in AI. However, certain limitations of the WEAT have been identified (ie, their nonrobust measure of bias and their reliance on predefined and limited groups of words or sentences), which may lead to inadequate measurements and evaluations of bias. Thus, this study takes a new approach at modifying this popular measure of bias, with a focus on making it more robust and applicable in other domains. In this study, we introduce the SD-WEAT, which is a modified version of the WEAT that uses the SD of multiple permutations of the WEATs to calculate bias in input embeddings. With the SD-WEAT, we evaluated the biases and stability of several language embedding models, including Global Vectors for Word Representation (GloVe), Word2Vec, and bidirectional encoder representations from transformers (BERT). This method produces results comparable to those of the WEAT, with strong correlations between the methods' bias scores or effect sizes (r=0.786) and P values (r=0.776), while addressing some of its largest limitations. More specifically, the SD-WEAT is more accessible, as it removes the need to predefine attribute groups, and because the SD-WEAT measures bias over multiple runs rather than one, it reduces the impact of outliers and sample size. Furthermore, the SD-WEAT was found to be more consistent and reliable than its predecessor. Thus, the SD-WEAT shows promise for robustly measuring bias in the input embeddings fed to AI language models.

IndoGovBERT: A Domain-Specific Language Model for Processing Indonesian Government SDG Documents

Achieving the Sustainable Development Goals (SDGs) requires collaboration among various stakeholders, particularly governments and non-state actors (NSAs). This collaboration results in but is also based on a continually growing volume of documents that needs to be analyzed and processed in a systematic way by government officials. Artificial Intelligence and Natural Language Processing (NLP) could, thus, offer valuable support for progressing towards SDG targets, including automating the government budget tagging and classifying NSA requests and initiatives, as well as helping uncover the possibilities for matching these two categories of activities. Many non-English speaking countries, including Indonesia, however, face limited NLP resources, such as, for instance, domain-specific pre-trained language models (PTLMs). This circumstance makes it difficult to automate document processing and improve the efficacy of SDG-related government efforts. The presented study introduces IndoGovBERT, a Bidirectional Encoder Representations from Transformers (BERT)-based PTLM built with domain-specific corpora, leveraging the Indonesian government’s public and internal documents. The model is intended to automate various laborious tasks of SDG document processing by the Indonesian government. Different approaches to PTLM development known from the literature are examined in the context of typical government settings. The most effective, in terms of the resultant model performance, but also most efficient, in terms of the computational resources required, methodology is determined and deployed for the development of the IndoGovBERT model. The developed model is then scrutinized in several text classification and similarity assessment experiments, where it is compared with four Indonesian general-purpose language models, a non-transformer approach of the Multilabel Topic Model (MLTM), as well as with a Multilingual BERT model. Results obtained in all experiments highlight the superior capability of the IndoGovBERT model for Indonesian government SDG document processing. The latter suggests that the proposed PTLM development methodology could be adopted to build high-performance specialized PTLMs for governments around the globe which face SDG document processing and other NLP challenges similar to the ones dealt with in the presented study.

Consumer Feedback Sentiment Classification Improved Via Modified Metaheuristic Optimization Natural Language Processing

This study investigates the synergy between the virtual and real-world economies through e-commerce, where seller reputation is critical in guiding consumer decisions. As traditional businesses shift towards online retail, user reviews become essential, offering feedback to both sellers and potential buyers. Sentiment analysis through machine learning (ML) techniques presents significant advantages for consumers and retailers alike. This research proposes a novel approach combining bidirectional encoder representations from transformers (BERT) embeddings with an optimized XGBoost classification model to enhance sentiment analysis performance. A modified metaheuristic algorithm, derived from the firefly algorithm (FA), is introduced to optimize the model. Testing on publicly available datasets demonstrates that models optimized by this algorithm achieved a peak accuracy of .881336. Further statistical analyses substantiate these improvements, and SHAP interpretation on the best-performing model identifies key features impacting model predictions, shedding light on factors driving customer sentiment insights.

Automated decision making in Barrett’s oesophagus: development and deployment of a natural language processing tool

Manual decisions regarding the timing of surveillance endoscopy for premalignant Barrett’s oesophagus (BO) is error-prone. This leads to inefficient resource usage and safety risks. To automate decision-making, we fine-tuned Bidirectional Encoder Representations from Transformers (BERT) models to categorize BO length (EndoBERT) and worst histopathological grade (PathBERT) on 4,831 endoscopy and 4,581 pathology reports from Guy’s and St Thomas’ Hospital (GSTT). The accuracies for EndoBERT test sets from GSTT, King’s College Hospital (KCH), and Sandwell and West Birmingham Hospitals (SWB) were 0.95, 0.86, and 0.99, respectively. Average accuracies for PathBERT were 0.93, 0.91, and 0.92, respectively. A retrospective analysis of 1640 GSTT reports revealed a 27% discrepancy between endoscopists’ decisions and model recommendations. This study underscores the development and deployment of NLP-based software in BO surveillance, demonstrating high performance at multiple sites. The analysis emphasizes the potential efficiency of automation in enhancing precision and guideline adherence in clinical decision-making.

A Chinese named entity recognition method for landslide geological disasters based on deep learning

Landslide Named Entity Recognition (LNER) involves extracting specific entities from Chinese unstructured landslide disaster texts, which is crucial for constructing a knowledge graph and supporting landslide prevention efforts. This study proposes a deep learning-based LNER model that utilizes Bidirectional Encoder Representations from Transformer (BERT) for word embeddings and integrates the Conditional Random Fields (CRF) algorithm and projected gradient descent (PGD) adversarial neural networks to enhance sequence labeling accuracy. The practical implications of this research lie in improving the efficiency and precision of disaster information extraction, aiding in real-time decision-making and risk mitigation strategies. Experiments on the constructed dataset show that the model effectively identifies eight types of landslide entities, achieving a highest F1 score of 89.7%.

A Text Generation Method Based on a Multimodal Knowledge Graph for Fault Diagnosis of Consumer Electronics

As consumer electronics evolve towards greater intelligence, their automation and complexity also increase, making it difficult for users to diagnose faults when they occur. To address the problem where users, relying solely on their own knowledge, struggle to diagnose faults in consumer electronics promptly and accurately, we propose a multimodal knowledge graph-based text generation method. Our method begins by using deep learning models like the Residual Network (ResNet) and Bidirectional Encoder Representations from Transformers (BERT) to extract features from user-provided fault information, which can include images, text, audio, and even olfactory data. These multimodal features are then combined to form a comprehensive representation. The fused features are fed into a graph convolutional network (GCN) for fault inference, identifying potential fault nodes in the electronics. These fault nodes are subsequently fed into a pre-constructed knowledge graph to determine the final diagnosis. Finally, this information is processed through the Bias-term Fine-tuning (BitFit) enhanced Chinese Pre-trained Transformer (CPT) model, which generates the final fault diagnosis text for the user. The experimental results show that our proposed method achieves a 4.4% improvement over baseline methods, reaching a fault diagnosis accuracy of 98.4%. Our approach effectively leverages multimodal fault information, addressing the challenges users face in diagnosing faults through the integration of graph convolutional network and knowledge graph technologies.

Electric Vehicle Sentiment Analysis Using Large Language Models

Sentiment analysis is a technique used to understand the public’s opinion towards an event, product, or organization. For example, sentiment analysis can be used to understand positive or negative opinions or attitudes towards electric vehicle (EV) brands. This provides companies with valuable insight into the public’s opinion of their products and brands. In the field of natural language processing (NLP), transformer models have shown great performance compared to traditional machine learning algorithms. However, these models have not been explored extensively in the EV domain. EV companies are becoming significant competitors in the automotive industry and are projected to cover up to 30% of the United States light vehicle market by 2030 In this study, we present a comparative study of large language models (LLMs) including bidirectional encoder representations from transformers (BERT), robustly optimised BERT (RoBERTa), and a generalised autoregressive pre-training method (XLNet) using Lucid Motors and Tesla Motors YouTube datasets. Results evidenced that LLMs like BERT and her variants are off-the-shelf algorithms for sentiment analysis, specifically when fine-tuned. Furthermore, our findings present the need for domain adaptation whilst utilizing LLMs. Finally, the experimental results showed that RoBERTa achieved consistent performance across the EV datasets with an F1 score of at least 92%.

Multifaceted Natural Language Processing Task-Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation.

The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model's comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non-English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes. In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents. Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks. The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F1-score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F1-score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41. This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications.

Evolution of the "Internet Plus Health Care" Mode Enabled by Artificial Intelligence: Development and Application of an Outpatient Triage System.

Although new technologies have increased the efficiency and convenience of medical care, patients still struggle to identify specialized outpatient departments in Chinese tertiary hospitals due to a lack of medical knowledge. The objective of our study was to develop a precise and subdividable outpatient triage system to improve the experiences and convenience of patient care. We collected 395,790 electronic medical records (EMRs) and 500 medical dialogue groups. The EMRs were divided into 3 data sets to design and train the triage model (n=387,876, 98%) and test (n=3957, 1%) and validate (n=3957, 1%) it. The triage system was altered based on the current BERT (Bidirectional Encoder Representations from Transformers) framework and evaluated by recommendation accuracies in Xinhua Hospital using the cancellation rates in 2021 and 2022, from October 29 to December 5. Finally, a prospective observational study containing 306 samples was conducted to compare the system's performance with that of triage nurses, which was evaluated by calculating precision, accuracy, recall of the top 3 recommended departments (recall@3), and time consumption. With 3957 (1%) records each, the testing and validation data sets achieved an accuracy of 0.8945 and 0.8941, respectively. Implemented in Xinhua Hospital, our triage system could accurately recommend 79 subspecialty departments and reduce the number of registration cancellations from 16,037 (3.83%) of the total 418,714 to 15,338 (3.53%) of the total 434200 (P<.05). In comparison to the triage system, the performance of the triage nurses was more accurate (0.9803 vs 0.9153) and precise (0.9213 vs 0.9049) since the system could identify subspecialty departments, whereas triage nurses or even general physicians can only recommend main departments. In addition, our triage system significantly outperformed triage nurses in recall@3 (0.6230 vs 0.5266; P<.001) and time consumption (10.11 vs 14.33 seconds; P<.001). The triage system demonstrates high accuracy in outpatient triage of all departments and excels in subspecialty department recommendations, which could decrease the cancellation rate and time consumption. It also improves the efficiency and convenience of clinical care to fulfill better the usage of medical resources, expand hospital effectiveness, and improve patient satisfaction in Chinese tertiary hospitals.

BiodiViz: Leveraging NER and RE for Automated Knowledge Graph Generation in Biodiversity Research

In biodiversity research, the integration of machine learning and data visualization is increasingly important for uncovering valuable insights from academic literature. This study introduces an innovative knowledge graph application, BiodiViz, designed to translate intricate text into intuitive visual representations, fostering a deeper comprehension of biodiversity relationships. BiodiViz uses the top-performing Named Entity Recognition (NER) and Relation Extraction (RE) models to automatically generate a comprehensive knowledge graph for biodiversity research. The NER model extracts and categorizes entities like organisms, phenomena, and habitats, while the RE model identifies relationships such as "have," "occur in," and "influence" from the BiodivNERE dataset (Abdelmageed et al. 2022). These entities and relationships are organized into nodes and edges within a graph. Researchers input text into BiodiViz, producing a visual knowledge graph that simplifies the analysis of complex biodiversity data, reducing manual effort and enhancing efficiency. Named Entity Recognition &amp; Relation Extraction BiodiViz leverages advanced Bidirectional Encoder Representations from Transformers (BERT)-based Large Language Models (LLMs) (Rogers et al. 2020), fine-tuned specifically for NER and RE tasks using the BiodivNERE dataset. The fine-tuning process involved various models, including BERT (Devlin et al. 2019), ELECTRA (Clark et al. 2020), and BiodivBERT (Abdelmageed et al. 2023). These models were evaluated for performance using the results of their F1-score as the main metric, which is the harmonic mean of precision (the proportion of true positive results among all positive predictions) and recall (the proportion of true positive results among all actual positives), with BiodivBERT achieving an F1-score of 77.16% for the NER task, while BERT excelled in the RE task with an F1-score of 81.28%. Rigorous hyperparameter optimization further enhanced the performance of BiodivBERT in the RE task by 3.38%. The BiodivNERE corpora by Abdelmageed et al. (2022) were used to fine-tune several models for NER and RE tasks in the biodiversity domain. The first corpus from the BiodivNERE corpora is BiodivNER, which is a gold standard dataset (manually labelled test corpora) for evaluating NER tasks. The fine-tuning process employed the token classification method from the Hugging Face library (Hugging Face 2023b), which assigns labels to each token in a sequence. Experiments were conducted with a batch size of four, meaning the model processes four examples/rows of data at a time before making an update to improve its learning. This is due to the constraints of the NVIDIA® GeForce RTX™ 3060 graphics processor. (NVIDIA 2024) Model performance was evaluated using the seqeval library (Nakayama 2018), focusing on accuracy, precision, recall, and F1 scores. For text classification, the second corpus, BiodivRE, was utilized, following previous research recommendations to explore fine-tuning settings for BiodivBERT. Hyperparameter optimization (Feurer and Hutter 2019) was conducted using Hugging Face’s Trainer API with an Optuna backend (Hugging Face 2023a), concentrating on learning rate and the number of training epochs (i.e., the number of complete passes through the entire dataset during model training). The BiodiViz Knowledge Graph Application The fine-tuned NER and RE models with the best F1-scores—BiodivBERT and BERT, respectively—were integrated into the knowledge graph application. Fig. 1 illustrates the flowchart of the application pipeline. Each sentence in the input text will go through the NER model to identify and label the entities within the sentence. Subsequently, these labeled entities, together with the original sentence, will be input into the RE model. The RE model will analyze every pair of entities for a potential relation and output the type of relation they share. The application will then utilize this data to create a graph with appropriate labels and color-coding. An example of the application's user interface with the knowledge graph is shown in Fig. 2. This study highlights the practical application of machine learning and data visualization in advancing biodiversity research, emphasizing the importance of developing user-friendly tools to support scientific exploration and discovery. The BiodiViz application, including the code and resources, is available on GitHub*1, providing an accessible tool for biodiversity researchers to streamline their analyses.

Identifying contextual content-based risk drivers for advanced risk management strategies

This research proposes a profound contextual topic identifier that incorporates topic modelling and a word embedding technique to discover and quantify corporate risks from its self-identified risk disclosures and examines the association between each risk type and operating performance via artificial intelligence (AI) technique. Via topic modelling adoption, we are able to discover the most essential risks confronted by corporates in the near future and evaluate how they respond to these risks. To gain deeper insight, the study performs a bidirectional encoder representation from transformers (BERT) (one type of word embedding approach) to extract and quantify the semantic features embedded into each risk disclosure. The results show that operating performance significantly and positively relates to corporate-specific risks. This study offers solid and direct support for authorities that set accounting principles to encourage firm managers to add a new section on risk factors in annual reports.