Bidirectional Encoder Representations From Transformers Research Articles

Multifaceted Natural Language Processing Task-Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation.

The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model's comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non-English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes. In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents. Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks. The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F1-score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F1-score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41. This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications.

Evolution of the "Internet Plus Health Care" Mode Enabled by Artificial Intelligence: Development and Application of an Outpatient Triage System.

Although new technologies have increased the efficiency and convenience of medical care, patients still struggle to identify specialized outpatient departments in Chinese tertiary hospitals due to a lack of medical knowledge. The objective of our study was to develop a precise and subdividable outpatient triage system to improve the experiences and convenience of patient care. We collected 395,790 electronic medical records (EMRs) and 500 medical dialogue groups. The EMRs were divided into 3 data sets to design and train the triage model (n=387,876, 98%) and test (n=3957, 1%) and validate (n=3957, 1%) it. The triage system was altered based on the current BERT (Bidirectional Encoder Representations from Transformers) framework and evaluated by recommendation accuracies in Xinhua Hospital using the cancellation rates in 2021 and 2022, from October 29 to December 5. Finally, a prospective observational study containing 306 samples was conducted to compare the system's performance with that of triage nurses, which was evaluated by calculating precision, accuracy, recall of the top 3 recommended departments (recall@3), and time consumption. With 3957 (1%) records each, the testing and validation data sets achieved an accuracy of 0.8945 and 0.8941, respectively. Implemented in Xinhua Hospital, our triage system could accurately recommend 79 subspecialty departments and reduce the number of registration cancellations from 16,037 (3.83%) of the total 418,714 to 15,338 (3.53%) of the total 434200 (P<.05). In comparison to the triage system, the performance of the triage nurses was more accurate (0.9803 vs 0.9153) and precise (0.9213 vs 0.9049) since the system could identify subspecialty departments, whereas triage nurses or even general physicians can only recommend main departments. In addition, our triage system significantly outperformed triage nurses in recall@3 (0.6230 vs 0.5266; P<.001) and time consumption (10.11 vs 14.33 seconds; P<.001). The triage system demonstrates high accuracy in outpatient triage of all departments and excels in subspecialty department recommendations, which could decrease the cancellation rate and time consumption. It also improves the efficiency and convenience of clinical care to fulfill better the usage of medical resources, expand hospital effectiveness, and improve patient satisfaction in Chinese tertiary hospitals.

BiodiViz: Leveraging NER and RE for Automated Knowledge Graph Generation in Biodiversity Research

In biodiversity research, the integration of machine learning and data visualization is increasingly important for uncovering valuable insights from academic literature. This study introduces an innovative knowledge graph application, BiodiViz, designed to translate intricate text into intuitive visual representations, fostering a deeper comprehension of biodiversity relationships. BiodiViz uses the top-performing Named Entity Recognition (NER) and Relation Extraction (RE) models to automatically generate a comprehensive knowledge graph for biodiversity research. The NER model extracts and categorizes entities like organisms, phenomena, and habitats, while the RE model identifies relationships such as "have," "occur in," and "influence" from the BiodivNERE dataset (Abdelmageed et al. 2022). These entities and relationships are organized into nodes and edges within a graph. Researchers input text into BiodiViz, producing a visual knowledge graph that simplifies the analysis of complex biodiversity data, reducing manual effort and enhancing efficiency. Named Entity Recognition &amp; Relation Extraction BiodiViz leverages advanced Bidirectional Encoder Representations from Transformers (BERT)-based Large Language Models (LLMs) (Rogers et al. 2020), fine-tuned specifically for NER and RE tasks using the BiodivNERE dataset. The fine-tuning process involved various models, including BERT (Devlin et al. 2019), ELECTRA (Clark et al. 2020), and BiodivBERT (Abdelmageed et al. 2023). These models were evaluated for performance using the results of their F1-score as the main metric, which is the harmonic mean of precision (the proportion of true positive results among all positive predictions) and recall (the proportion of true positive results among all actual positives), with BiodivBERT achieving an F1-score of 77.16% for the NER task, while BERT excelled in the RE task with an F1-score of 81.28%. Rigorous hyperparameter optimization further enhanced the performance of BiodivBERT in the RE task by 3.38%. The BiodivNERE corpora by Abdelmageed et al. (2022) were used to fine-tune several models for NER and RE tasks in the biodiversity domain. The first corpus from the BiodivNERE corpora is BiodivNER, which is a gold standard dataset (manually labelled test corpora) for evaluating NER tasks. The fine-tuning process employed the token classification method from the Hugging Face library (Hugging Face 2023b), which assigns labels to each token in a sequence. Experiments were conducted with a batch size of four, meaning the model processes four examples/rows of data at a time before making an update to improve its learning. This is due to the constraints of the NVIDIA® GeForce RTX™ 3060 graphics processor. (NVIDIA 2024) Model performance was evaluated using the seqeval library (Nakayama 2018), focusing on accuracy, precision, recall, and F1 scores. For text classification, the second corpus, BiodivRE, was utilized, following previous research recommendations to explore fine-tuning settings for BiodivBERT. Hyperparameter optimization (Feurer and Hutter 2019) was conducted using Hugging Face’s Trainer API with an Optuna backend (Hugging Face 2023a), concentrating on learning rate and the number of training epochs (i.e., the number of complete passes through the entire dataset during model training). The BiodiViz Knowledge Graph Application The fine-tuned NER and RE models with the best F1-scores—BiodivBERT and BERT, respectively—were integrated into the knowledge graph application. Fig. 1 illustrates the flowchart of the application pipeline. Each sentence in the input text will go through the NER model to identify and label the entities within the sentence. Subsequently, these labeled entities, together with the original sentence, will be input into the RE model. The RE model will analyze every pair of entities for a potential relation and output the type of relation they share. The application will then utilize this data to create a graph with appropriate labels and color-coding. An example of the application's user interface with the knowledge graph is shown in Fig. 2. This study highlights the practical application of machine learning and data visualization in advancing biodiversity research, emphasizing the importance of developing user-friendly tools to support scientific exploration and discovery. The BiodiViz application, including the code and resources, is available on GitHub*1, providing an accessible tool for biodiversity researchers to streamline their analyses.

Evaluation of rural tourism development level using BERT-enhanced deep learning model and BP algorithm

To address the insufficient expressive capabilities of traditional methods in assessing the development level of rural tourism, this study explores the fusion application of the Bidirectional Encoder Representations from Transformers (BERT) and the Back Propagation (BP) algorithm to enhance the accuracy and comprehensiveness of rural tourism development assessment. Firstly, this study introduces the BERT deep learning model and its applications in natural language processing, alongside the role of the BP algorithm in pattern recognition and predictive analysis. Subsequently, a framework for assessing rural tourism development levels, integrating BERT and the BP algorithm, is proposed. This framework collects multidimensional rural tourism-related data and utilizes the BERT model for sentiment analysis and topic extraction from textual data. Empirical analysis of rural tourism development in a specific region validates the effectiveness of the proposed approach. Experimental results demonstrate: (1) The model achieves an accuracy of 84.33% and an F1 score of 85.33% on the publicly available Laptop dataset, with a processing time of 20 s, significantly outperforming other methods. Compared to traditional approaches, the proposed method accurately captures correlations between textual information and numerical data, thereby enhancing the credibility and accuracy of assessment results. (2) From the ablation study results, it is evident that removing any component from the model leads to performance degradation. Specifically, removing the Bidirectional Gated Recurrent Unit (BiGRU) reduces accuracy and F1 scores to 78.21% and 76.33% on the Laptop dataset, and to 85.10% and 70.45% on the Tourist_F dataset. Removing Text Convolutional Neural Network (CNN) reduces accuracy and F1 scores to 79.34% and 77.56% on the Laptop dataset, and to 86.25% and 72.11% on the Tourist_F dataset. The most significant performance decline occurs upon removing BERT, with accuracy and F1 scores decreasing to 70.14% and 66.43% on the Laptop dataset, and to 78.45% and 60.33% on the Tourist_F dataset. These results indicate that BiGcRU, TextCNN, and BERT each contribute significantly to the model’s performance. This study provides substantial support for advancing sustainable development in rural tourism, offering significant practical and innovative value.

A deep learning model for syncope recognition in the emergency department

Abstract Introduction Syncope is a common presenting symptom in the emergency departments (EDs) and poses challenges in its recognition and differentiation from alternative causes of transient loss of consciousness. Natural Language Processing (NLP) is an artificial intelligence technique that enables the analysis of natural language, such as clinical notes and medical history within electronic medical records (EMRs). The application of NLP into the medical field is surging but its use for the recognition of syncope is still underexplored. Purpose We aimed to develop a NLP-based deep learning model for syncope recognition in the ED. Methods Two models, based on the Italian and Multilingual Bidirectional Encoder Representations from Transformers (BERT) language model, were developed and evaluated using consecutive EMRs from Humanitas Research Hospital ED. The first "triage" model was only based on textual data contained in the "triage" section of the EMR. The second "anamnesis" model also considered textual data contained in the "medical history" section. Performance metrics including area under the curve (AUC), Matthew’s correlation coefficient (MCC) and Brier’s score (BS) were used to compare the discriminative capabilities of different BERT models. Results The models were developed and tested on 15,098 and 15,222 EMRs, respectively. The Multilanguage BERT discriminated patients with syncope with an AUC of 0.85 in the triage model and of 0.92 in the anamnesis model. The Italian BERT identified patients with syncope with a MCC of 0.50 and a BS of 0.02 in the triage model and of 0.64 and 0.01, respectively, in the anamnesis model. Conclusions We propose two NLP-based models that identify syncope patients with a high discriminative capability. The presented algorithm may provide a starting point for the management of syncope patients using machine learning techniques.ROC and calibration curves of modelsFramework for model implementation

Unveiling User Sentiment: Aspect-Based Analysis and Topic Modeling of Ride-Hailing and Google Play App Reviews

Background: Mobile app usage is increasing in the digital age, with Ride-Hailing app becoming the primary example of this trend. To obtain valuable understanding of how people perceive and interact with mobile app, user reviews on platforms such as Google Play are usually analyzed. This analysis can assist developers to identify areas for improvement in both Ride-hailing and Google Play App. A promising method that can be used to analyze user perception in this instance is Aspect-Based Sentiment Analysis (ABSA). Objective: This research aimed to apply ABSA to user reviews using Bidirectional Encoder Representations from Transformers (BERT) models. In this context, aspect identification and topic modeling were performed by using Latent Dirichlet Allocation (LDA). The model extracted topics from the reviews and used Generative Artificial Intelligence (GenAI) to define the aspects of the topics to further enhance the analysis. For consistency and accuracy, the method included sentiment annotation by a human annotator. Methods: A total of two datasets were used in this research, with the first collected by scraping user reviews of Ride-Hailing App while the second was obtained from Kaggle, and to identify relevant topics, modeling was performed using LDA. These topics were then categorized into aspects using GenAI, covering areas, such as customer experience, service, payment, app features, task management, and event management. Subsequently, sentiment labeling was conducted using human annotators to provide a reliable baseline. BERT model was then used to classify sentiment with aspect hints, and the evaluation included calculations of accuracy, precision, recall, and F1-score. Results: The results showed that BERT model achieved the highest accuracy of 97% in sentiment analysis across all datasets. Conclusion: This research provided valuable understanding of user experience and established a strong ABSA framework for analyzing user reviews using LDA, Aspect Annotation, GenAI, and BERT sentiment models. Future research could expand this method to other app categories and incorporate real-time ABSA for continuous monitoring and dynamic feedback. Keywords: User Reviews, Aspect-Based Sentiment Analysis (ABSA), Sentiment Analysis, Topic Modeling, Generative Artificial Intelligence (GenAI)

Dual-tiered insights: cross-examining entities in free text electronic health records

Abstract Background Poor determination and processing of actions at the primary care level are found to be associated with unstructured and cluttered electronic health records (EHRs). Physicians recommend that EHRs must highlight entities, such as action items, findings, follow-ups, and reasons for medication changes. There’s consensus around the success of large language models (LLMs) in detecting clinical and non-clinical entities in EHRs. However, end-to-end systems make limited efforts to go beyond entity and relationship detection to support clinical decision-making. Methods We propose a two-tier system that generates case-based comparative reports for entities highlighted by physicians in EHRs. First, we fine-tuned a custom Bidirectional Encoder Representations from Transformers (BERT) model that detects the required entities within the EHRs. Second, a clustering-based service generates a report comparing the detected entities of a specific EHR with other EHRs sharing the same conditions determined by the healthcare professional. Results The system leverages the MIMIC IV deidentified free-text clinical notes dataset. The first-tier successfully classified entities with a macro-score average range of [0.87 - 0.94]. For the second tier, preliminary simulations on a set of test samples were made to cluster the entities. The simulations leverage a custom statistical tests engine offering a suite of entity-type specific tests. The results of these tests are used to generate the case-based detected entities comparative report. Conclusions Highlighting clinical and non-clinical entities bridges the communication gap between primary and secondary care. Clustering these entities extends the capability of LLMs in clinical support decision systems by providing healthcare professionals with the ability to run comparative analyses between EHRs. The comparative analysis reports provide a factual comparison between a selected number of detected entities and their thresholds. Key messages • Clustering clinical and non-clinical entities extends the capacity of named entity recognition in clinical decision-making systems. • Ability to generate a comparative analysis report based on figures extracted from recognised entities in free-text electronic health records.

Chinese Public Attitudes and Opinions on Health Policies During Public Health Emergencies: Sentiment and Topic Analysis.

By the end of 2021, the new wave of COVID-19 sparked by the Omicron variant spread rapidly due to its highly contagious nature, affecting more than 170 countries worldwide. Nucleic acid testing became the gold standard for diagnosing novel coronavirus infections. As of July 2022, numerous cities and regions in China have implemented regular nucleic acid testing policies, which have had a significant impact on socioeconomics and people's lives. This policy has garnered widespread attention on social media platforms. This study took the newly issued regular nucleic acid testing policy during the COVID-19 pandemic as an example to explore the sentiment responses and fluctuations of netizens toward new policies during public health emergencies. It aimed to propose strategies for managing public opinion on the internet and provide recommendations for policy making and public opinion control. We collected blog posts related to nucleic acid testing on Weibo from April 1, 2022, to July 31, 2022. We used the topic modeling technique latent Dirichlet allocation (LDA) to identify the most common topics posted by users. We used Bidirectional Encoder Representations from Transformers (BERT) to calculate the sentiment score of each post. We used an autoregressive integrated moving average (ARIMA) model to examine the relationship between sentiment scores and changes over time. We compared the differences in sentiment scores across various topics, as well as the changes in sentiment before and after the announcement of the nucleic acid price reduction policy (May 22) and the lifting of the lockdown policy in Shanghai (June 1). We collected a total of 463,566 Weibo posts, with an average of 3799.72 (SD 1296.06) posts published daily. The LDA topic extraction identified 8 topics, with the most numerous being the Shanghai outbreak, nucleic acid testing price, and transportation. The average sentiment score of the posts was 0.64 (SD 0.31), indicating a predominance of positive sentiment. For all topics, posts with positive sentiment consistently outnumbered those with negative sentiment (χ27=24,844.4, P<.001). The sentiment scores of posts related to "nucleic acid testing price" decreased after May 22 compared with before (t120=3.882, P<.001). Similarly, the sentiment scores of posts related to the "Shanghai outbreak" decreased after June 1 compared with before (t120=11.943, P<.001). During public health emergencies, the topics of public concern were diverse. Public sentiment toward the regular nucleic acid testing policy was generally positive, but fluctuations occurred following the announcement of key policies. To understand the primary concerns of the public, the government needs to monitor social media posts by citizens. By promptly sharing information on media platforms and engaging in effective communication, the government can bridge the information gap between the public and government agencies, fostering a positive public opinion environment.

Capturing multiple emotions from conversational data using fine tuned transformers

Emotion detection is one of the crucial topics of Natural Language Processing (NLP) in recent years, and now one of the biggest motivating factors in correct identification and interpretation of a wide range of emotional expressions in textual data. This research examines the use of Bidirectional Encoder Representations from Transformers (BERT), a trained transformer model for emotion detection in textual data. This analysis evaluates how good BERT is at identifying emotions such as surprise, anger, fear, happiness and sadness compared with ordinary machine learning as well as deep learning techniques. The addition of weighted emotions approach enhances the model performance and gives a deeper emotional context awareness making it more effective to deal with complicated emotional utterances. The purpose of the research is to develop a fine-tuned BERT model with a weighted emotion framework to enhance the accuracy of emotion classification in conversational text. The context is improving emotion recognition in scenarios where multiple emotions co-exist, addressing limitations in traditional models by capturing subtle and overlapping emotional expressions. In terms of training the methods multiple datasets are considered and also the research examines various models performance. The article further discusses possible application areas of BERT modified in light of NLP.

TECRR: a benchmark dataset of radiological reports for BI-RADS classification with machine learning, deep learning, and large language model baselines

BackgroundRecently, machine learning (ML), deep learning (DL), and natural language processing (NLP) have provided promising results in the free-form radiological reports’ classification in the respective medical domain. In order to classify radiological reports properly, a high-quality annotated and curated dataset is required. Currently, no publicly available breast imaging-based radiological dataset exists for the classification of Breast Imaging Reporting and Data System (BI-RADS) categories and breast density scores, as characterized by the American College of Radiology (ACR). To tackle this problem, we construct and annotate a breast imaging-based radiological reports dataset and its benchmark results. The dataset was originally in Spanish. Board-certified radiologists collected and annotated it according to the BI-RADS lexicon and categories at the Breast Radiology department, TecSalud Hospitals Monterrey, Mexico. Initially, it was translated into English language using Google Translate. Afterwards, it was preprocessed by removing duplicates and missing values. After preprocessing, the final dataset consists of 5046 unique reports from 5046 patients with an average age of 53 years and 100% women. Furthermore, we used word-level NLP-based embedding techniques, term frequency-inverse document frequency (TF-IDF) and word2vec to extract semantic and syntactic information. We also compared the performance of ML, DL and large language models (LLMs) classifiers for BI-RADS category classification.ResultsThe final breast imaging-based radiological reports dataset contains 5046 unique reports. We compared K-Nearest Neighbour (KNN), Support Vector Machine (SVM), Naive Bayes (NB), Random Forest (RF), Adaptive Boosting (AdaBoost), Gradient-Boosting (GB), Extreme Gradient Boosting (XGB), Long Short-Term Memory (LSTM), Bidirectional Encoder Representations from Transformers (BERT) and Biomedical Generative Pre-trained Transformer (BioGPT) classifiers. It is observed that the BioGPT classifier with preprocessed data performed 6% better with a mean sensitivity of 0.60 (95% confidence interval (CI), 0.391-0.812) compared to the second best performing classifier BERT, which achieved mean sensitivity of 0.54 (95% CI, 0.477-0.607).ConclusionIn this work, we propose a curated and annotated benchmark dataset that can be used for BI-RADS and breast density category classification. We also provide baseline results of most ML, DL and LLMs models for BI-RADS classification that can be used as a starting point for future investigation. The main objective of this investigation is to provide a repository for the investigators who wish to enter the field to push the boundaries further.

A novel integrated prediction method using adaptive mode decomposition, attention mechanism and deep learning for coking products prices

Accurate prediction of coking product prices is crucial for enhancing production efficiency, cost optimization, and profit maximization in smart coking facilities. To address the volatility caused by nonlinear factors such as raw material costs, substitutes, macroeconomic indicators, sudden events, policy changes, and market behaviors, we propose a novel integrated prediction method for coking product price prediction. This method combines Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) for signal decomposition, Bidirectional Encoder Representations from Transformers (BERT) for natural language processing, attention mechanisms (AT) to weigh feature importance, and an ensemble of Bidirectional Gated Recurrent Unit, Bidirectional Long Short-Term Memory, and Gated Recurrent Unit, abbreviated BBG, for robust feature extraction. We design a feature selection strategy to avoid data leakage and improve the predictive ability of the model, and describe a method to maintain textual data information integrity when combining data from different sources. Experimental results on coke and methanol datasets show that our approach retains multi-source text richness improves predictive capability, and outperforms other state-of-the-art methods, providing an effective tool for developing smart coke plants.

Assessing Scientific Text Similarity: A Novel Approach Utilizing Non-Negative Matrix Factorization and Bidirectional Encoder Representations from Transformer

The patent serves as a vital component of scientific text, and over time, escalating competition has generated a substantial demand for patent analysis encompassing areas such as company strategy and legal services, necessitating fast, accurate, and easily applicable similarity estimators. At present, conducting natural language processing(NLP) on patent content, including titles, abstracts, etc., can serve as an effective method for estimating similarity. However, the traditional NLP approach has some disadvantages, such as the requirement for a huge amount of labeled data and poor explanation of deep-learning-based model internals, exacerbated by the high compression of patent content. On the other hand, most knowledge-based deep learning models require a vast amount of additional analysis results as training variables in similarity estimation, which are limited due to human participation in the analysis part. Thus, in this research, addressing these challenges, we introduce a novel estimator to enhance the transparency of similarity estimation. This approach integrates a patent’s content with international patent classification (IPC), leveraging bidirectional encoder representations from transformers (BERT), and non-negative matrix factorization (NMF). By integrating these techniques, we aim to improve knowledge discovery transparency in NLP across various IPC dimensions and incorporate more background knowledge into context similarity estimation. The experimental results demonstrate that our model is reliable, explainable, highly accurate, and practically usable.

Enhanced BERT-based Multi-Head Self-Attention Transformer for Transformation of Marathi Text to Marathi Sign Language Gloss

One recent advancement in the field of machine learning is the translation of text into sign language gloss, which is a form of natural language for the deaf community. The work presented is a new method to translate Marathi text to Marathi sign language gloss by combining salient features of Bidirectional Encoder Representation from Transformer (BERT) for tokenization and complementing the tokenized frame with Encoder with Attention mechanism and decoding with the LSTM decoder. The work conducted experiments on the created corpora of Marathi text and Marathi sign language gloss sentence pairs. The experiments that employed three models show that the suggested model performs better than the existing approaches. The results show that the translation of Marathi text to sign gloss achieves improved performances with an accuracy of 91.5% and the BLEU scores BLEU-1: 85, BLEU-2: 75, BLEU-3: 65, and BLEU-4: 57.

Advanced neural network-based model for predicting court decisions on child custody

Predicting court rulings has gained attention over the past years. The court rulings are among the most important documents in all legal systems, profoundly impacting the lives of the children in case of divorce or separation. It is evident from literature that Natural language processing (NLP) and machine learning (ML) are widely used in the prediction of court rulings. In general, the court decisions comprise several pages and require a lot of space. In addition, extracting valuable information and predicting legal decisions task is difficult. Moreover, the legal system’s complexity and massive litigation make this problem more serious. Thus to solve this issue, we propose a new neural network-based model for predicting court decisions on child custody. Our proposed model efficiently performs an efficient search from a massive court decisions database and accurately identifies specific ones that especially deal with copyright claims. More specially, our proposed model performs a careful analysis of court decisions, especially on child custody, and pinpoints the plaintiff’s custody request, the court’s ruling, and the pivotal arguments. The working mechanism of our proposed model is performed in two phases. In the first phase, the isolation of pertinent sentences within the court ruling encapsulates the essence of the proceedings performed. In the second phase, these documents were annotated independently by using two legal professionals. In this phase, NLP and transformer-based models were employed and thus processed 3,000 annotated court rulings. We have used a massive dataset for the training and refining of our proposed model. The novelty of the proposed model is the integration of bidirectional encoder representations from transformers (BERT) and bidirectional long short-term memory (Bi_LSTM). The traditional methods are primarily based on support vector machines (SVM), and logistic regression. We have performed a comparison with the state-of-the-art model. The efficient results indicate that our proposed model efficiently navigates the complex terrain of legal language and court decision structures. The efficiency of the proposed model is measured in terms of the F1 score. The achieved results show that scores range from 0.66 to 0.93 and Kappa indices from 0.57 to 0.80 across the board. The performance is achieved at times surpassing the inter-annotator agreement, underscoring the model’s adeptness at extracting and understanding nuanced legal concepts. The efficient results proved the potential of the proposed neural network model, particularly those based on transformers, to effectively discern and categorize key elements within legal texts, even amidst the intricacies of judicial language and the layered complexity of appellate rulings.

Perceived Responses of International Tourists to Transportation and Tourism Services During Typhoons Faxai and Hagibis in Japan

There is a limited understanding on the information-seeking behavior of international tourists during disaster response scenarios due to the lack of empirical studies on crisis communication in Japan. This study clarifies the topics generated from both international tourists and official Twitter accounts by applying the embedding Bidirectional Encoder Representations from Transformers (BERT) topic model and examines the temporal sentiment changes toward transportation and tourism using the sentiment scores obtained from topic-based Valence Aware Dictionary and sEntiment Reasoner (VADER) sentiment analysis during disasters. A case study was conducted using Twitter data on Typhoons Faxai and Hagibis, which struck Japan in 2019. This study found differences in the topics generated among international tourists and officials in response and a continuous negative sentiment toward specific transportation services. The managerial implications of these findings regarding the use of social media in crisis communication in tourism are also discussed.

Can the sentiment of the official media predict the return volatility of the Chinese crude oil futures?

This study investigates whether the sentiment of Chinese official media towards crude oil influences price volatility of the Chinese crude oil futures (SC). By leveraging textual analysis through Bidirectional Encoder Representations from Transformers (BERT), we quantify the sentiment of oil-related articles published by the primary official media in China. Our main finding, building on both in-sample and out-of-sample analyses, robustly reveals that this sentiment significantly forecasts the one-day-ahead intraday return volatility of SC. Moreover, we extend the analysis to different time horizons (i.e., one-week and one-month-ahead) and find the prominent forecasting power of the official media sentiment as well. We also find that the official media sentiment fails to forecast the price volatility of WTI oil futures, implying that the official media sentiment contains some unique Chinese information. Overall, our study contributes to the existing literature on predicting the return volatility of the Chinese crude oil futures, and offers fresh insights into an essential yet underexplored sentiment, i.e., official media sentiment.

Design of agricultural question answering information extraction method based on improved BILSTM algorithm

With the rapid growth of the agricultural information and the need for data analysis, how to accurately extract useful information from massive data has become an urgent first step in agricultural data mining and application. In this study, an agricultural question-answering information extraction method based on the BE-BILSTM (Improved Bidirectional Long Short-Term Memory) algorithm is designed. Firstly, it uses Python’s Scrapy crawler framework to obtain the information of soil types, crop diseases and pests, and agricultural trade information, and remove abnormal values. Secondly, the information extraction converts the semi-structured data by using entity extraction methods. Thirdly, the BERT (Bidirectional Encoder Representations from Transformers) algorithm is introduced to improve the performance of the BILSTM algorithm. After comparing with the BERT-CRF (Conditional Random Field) and BILSTM algorithm, the result shows that the BE-BILSTM algorithm has better information extraction performance than the other two algorithms. This study improves the accuracy of the agricultural information recommendation system from the perspective of information extraction. Compared with other work that is done from the perspective of recommendation algorithm optimization, it is more innovative; it helps to understand the semantics and contextual relationships in agricultural question and answer, which improves the accuracy of agricultural information recommendation systems. By gaining a deeper understanding of farmers’ needs and interests, the system can better recommend relevant and practical information.

Evaluating Medical Entity Recognition in Health Care: Entity Model Quantitative Study.

Named entity recognition (NER) models are essential for extracting structured information from unstructured medical texts by identifying entities such as diseases, treatments, and conditions, enhancing clinical decision-making and research. Innovations in machine learning, particularly those involving Bidirectional Encoder Representations From Transformers (BERT)-based deep learning and large language models, have significantly advanced NER capabilities. However, their performance varies across medical datasets due to the complexity and diversity of medical terminology. Previous studies have often focused on overall performance, neglecting specific challenges in medical contexts and the impact of macrofactors like lexical composition on prediction accuracy. These gaps hinder the development of optimized NER models for medical applications. This study aims to meticulously evaluate the performance of various NER models in the context of medical text analysis, focusing on how complex medical terminology affects entity recognition accuracy. Additionally, we explored the influence of macrofactors on model performance, seeking to provide insights for refining NER models and enhancing their reliability for medical applications. This study comprehensively evaluated 7 NER models-hidden Markov models, conditional random fields, BERT for Biomedical Text Mining, Big Transformer Models for Efficient Long-Sequence Attention, Decoding-enhanced BERT with Disentangled Attention, Robustly Optimized BERT Pretraining Approach, and Gemma-across 3 medical datasets: Revised Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA), BioCreative V CDR, and Anatomical Entity Mention (AnatEM). The evaluation focused on prediction accuracy, resource use (eg, central processing unit and graphics processing unit use), and the impact of fine-tuning hyperparameters. The macrofactors affecting model performance were also screened using the multilevel factor elimination algorithm. The fine-tuned BERT for Biomedical Text Mining, with balanced resource use, generally achieved the highest prediction accuracy across the Revised JNLPBA and AnatEM datasets, with microaverage (AVG_MICRO) scores of 0.932 and 0.8494, respectively, highlighting its superior proficiency in identifying medical entities. Gemma, fine-tuned using the low-rank adaptation technique, achieved the highest accuracy on the BioCreative V CDR dataset with an AVG_MICRO score of 0.9962 but exhibited variability across the other datasets (AVG_MICRO scores of 0.9088 on the Revised JNLPBA and 0.8029 on AnatEM), indicating a need for further optimization. In addition, our analysis revealed that 2 macrofactors, entity phrase length and the number of entity words in each entity phrase, significantly influenced model performance. This study highlights the essential role of NER models in medical informatics, emphasizing the imperative for model optimization via precise data targeting and fine-tuning. The insights from this study will notably improve clinical decision-making and facilitate the creation of more sophisticated and effective medical NER models.

Zero-Shot Learning for Accurate Project Duration Prediction in Crowdsourcing Software Development

Crowdsourcing Software Development (CSD) platforms, i.e., TopCoder, function as intermediaries connecting clients with developers. Despite employing systematic methodologies, these platforms frequently encounter high task abandonment rates, with approximately 19% of projects failing to meet satisfactory outcomes. Although existing research has focused on task scheduling, developer recommendations, and reward mechanisms, there has been insufficient attention to the support of platform moderators, or copilots, who are essential to project success. A critical responsibility of copilots is estimating project duration; however, manual predictions often lead to inconsistencies and delays. This paper introduces an innovative machine learning approach designed to automate the prediction of project duration on CSD platforms. Utilizing historical data from TopCoder, the proposed method extracts pertinent project attributes and preprocesses textual data through Natural Language Processing (NLP). Bidirectional Encoder Representations from Transformers (BERT) are employed to convert textual information into vectors, which are then analyzed using various machine learning algorithms. Zero-shot learning algorithms exhibit superior performance, with an average accuracy of 92.76%, precision of 92.76%, recall of 99.33%, and an f-measure of 95.93%. The implementation of the proposed automated duration prediction model is crucial for enhancing the success rate of crowdsourcing projects, optimizing resource allocation, managing budgets effectively, and improving stakeholder satisfaction.

Investigating the agenda of global warming on Twitter: A machine learning approach

This study investigates the agenda of global warming issue based on the two levels of agenda-setting framework. Using the Bidirectional Encoder Representations from Transformers (BERT)-based models, it identifies nine attributes of global warming issue and estimates the average tone of tweets in each month of 2021. The findings suggest the attributes are interconnected and partisan divides existed to a larger extent in certain attributes than in others. The nine attributes identified, their inherent interconnections as well as different tones associated with these attributes have the implications for policy-makers to formulate and evaluate policies regarding this multifaceted issue.