Deep Learning-based Natural Language Processing Research Articles

How to strategically respond to online hotel reviews: A strategy-aware deep learning approach

Online reviews exert a considerable influence on consumer purchase behavior, yet there remains ambiguity about the most effective managerial response strategies for positive and negative reviews. Addressing this gap, our study introduces a Strategy-Aware, Deep Learning-Based Natural Language Processing (Sa-DLNLP) model designed to optimize firm responses. The proposed model underwent rigorous evaluation through a human-coded study and was subsequently validated by a separate user response study. Our findings reveal that active-constructive responses significantly enhance the impact of positive reviews, whereas passive-constructive strategies are more effective in mitigating the damage from negative reviews. Additionally, the study underscores the importance of concise, personalized, and prompt responses across the board. Interestingly, responses that are overly explanatory, excessively empathetic, or challenge customers were found to be counterproductive when dealing with negative reviews. This study not only demystifies the art of managing online reviews but also offers an advanced deep learning methodology that can directly benefit the disciplines of Information Systems and Management.

Optimisation Study of Natural Language Processing Algorithms Based on Deep Learning

This paper presents a comprehensive optimization study of natural language processing (NLP) algorithms based on deep learning techniques. The research explores various strategies to enhance the performance and efficiency of NLP models, aiming to address the challenges posed by large-scale datasets and complex linguistic structures. Through a systematic review of existing literature and methodologies, this study synthesizes insights into key optimization approaches, including model architecture design, parameter tuning, and data preprocessing techniques. Moreover, it investigates the impact of different optimization strategies on NLP tasks such as sentiment analysis, named entity recognition, and machine translation. By elucidating the strengths and limitations of various optimization techniques, this paper offers valuable insights for researchers and practitioners in the field of deep learning-based NLP.

Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triage

ObjectiveThe manual recording of electronic health records (EHRs) by clinicians in the emergency department (ED) is time-consuming and challenging. In light of recent advancements in large language models (LLMs) such as GPT and BERT, this study aimed to design and validate LLMs for automatic clinical diagnoses. The models were designed to identify 12 medical symptoms and 2 patient histories from simulated clinician–patient conversations within 6 primary symptom scenarios in emergency triage rooms. Materials and methodWe developed classification models by fine-tuning BERT, a transformer-based pre-trained model. We subsequently analyzed these models using eXplainable artificial intelligence (XAI) and the Shapley additive explanation (SHAP) method. A Turing test was conducted to ascertain the reliability of the XAI results by comparing them to the outcomes of tasks performed and explained by medical workers. An emergency medicine specialist assessed the results of both XAI and the medical workers. ResultsWe fine-tuned four pre-trained LLMs and compared their classification performance. The KLUE-RoBERTa-based model demonstrated the highest performance (F1-score: 0.965, AUROC: 0.893) on human-transcribed script data. The XAI results using SHAP showed an average Jaccard similarity of 0.722 when compared with explanations of medical workers for 15 samples. The Turing test results revealed a small 6% gap, with XAI and medical workers receiving the mean scores of 3.327 and 3.52, respectively. ConclusionThis paper highlights the potential of LLMs for automatic EHR recording in Korean EDs. The KLUE-RoBERTa-based model demonstrated superior classification performance. Furthermore, XAI using SHAP provided reliable explanations for model outputs. The reliability of these explanations was confirmed by a Turing test.

미디어 텍스트 분석 기반의 공급망 리스크 모니터링 시스템의 개발

Recently, companies are exposed to various supply chain risks such as intensified trade conflicts, epidemics, economic and geopolitical uncertainties, and natural disasters. Thus there is increasing importance in monitoring information related to supply chain risks. Analyzing real-time media texts, such as news articles, can be utilized for monitoring up-to-date information supply chain risks. However, researches regarding analyzing supply chain risk related text are in early stages, and researches to apply modern AI techniques such as deep learning-based natural language processing to supply chain risk texts are scarce. This study aims to develop a supply chain risk monitoring system that monitors and extracts information related to supply chain risks by analyzing news articles. To collect supply chain risk related articles a filtering model based on KoBERT is developed, of which risk types are identified based on LDA topic modeling to be utilized as the train data. To predict news articles’ risk types, two deep learning- based risk classification models are developed using BOW(Bag of Words) and KoBERT. The results showed high accuracy of KoBERT based model in filtering supply chain risk-related articles, and in the classification of supply chain risk types also KoBERT based model showed better performance than BOW based model.

IDPpub: Illuminating the Dark Phosphoproteome Through PubMed Mining

Global phosphoproteomics experiments quantify tens of thousands of phosphorylation sites. However, data interpretation is hampered by our limited knowledge on functions, biological contexts, or precipitating enzymes of the phosphosites. This study establishes a repository of phosphosites with associated evidence in biomedical abstracts, using deep learning-based natural language processing techniques. Our model for illuminating the dark phosphoproteome through PubMed mining (IDPpub) was generated by fine-tuning BioBERT, a deep learning tool for biomedical text mining. Trained using sentences containing protein substrates and phosphorylation site positions from 3000 abstracts, the IDPpub model was then used to extract phosphorylation sites from all MEDLINE abstracts. The extracted proteins were normalized to gene symbols using the National Center for Biotechnology Information gene query, and sites were mapped to human UniProt sequences using ProtMapper and mouse UniProt sequences by direct match. Precision and recall were calculated using 150 curated abstracts, and utility was assessed by analyzing the CPTAC (Clinical Proteomics Tumor Analysis Consortium) pan-cancer phosphoproteomics datasets and the PhosphoSitePlus database. Using 10-fold cross validation, pairs of correct substrates and phosphosite positions were extracted with an average precision of 0.93 and recall of 0.94. After entity normalization and site mapping to human reference sequences, an independent validation achieved a precision of 0.91 and recall of 0.77. The IDPpub repository contains 18,458 unique human phosphorylation sites with evidence sentences from 58,227 abstracts and 5918 mouse sites in 14,610 abstracts. This included evidence sentences for 1803 sites identified in CPTAC studies that are not covered by manually curated functional information in PhosphoSitePlus. Evaluation results demonstrate the potential of IDPpub as an effective biomedical text mining tool for collecting phosphosites. Moreover, the repository (http://idppub.ptmax.org), which can be automatically updated, can serve as a powerful complement to existing resources.

Abstract 16207: Deep Learning-Based Natural Language Processing of Discharge Summaries for Automated Identification of Heart Failure With Reduced Ejection Fraction

Background: Implementation of a national program to improve guideline-directed care in heart failure with reduced ejection fraction (HFrEF) has been limited by the lack of automated tools to measure care quality. A key challenge for this approach is to identify patients with HFrEF at hospital discharge, an opportunity to evaluate and improve care. Methods: We developed a novel deep-learning language model to detect patients with HFrEF from discharge summaries using a semi-supervised learning framework. Hospitalizations with heart failure (HF) at Yale New Haven Hospital between 2015-2019 were labeled as HFrEF if left ventricular ejection fraction was under 40% on antecedent echocardiography. The model was internally validated with model-based net reclassification improvement (NRI) assessed against chart-based diagnosis codes. We externally validated the model on discharge summaries from hospitalizations with HF at Northwestern Medicine, community hospitals of Yale Health, and MIMIC-III database, confirmed with chart abstraction. Results: A total of 13,251 notes from 5,392 individuals (age 73 ± 14 years, 48% female), including 2,487 (46.1%) patients with HFrEF were used for model development (train/validation/test: 70/15/15%). The model achieved an AUROC 0.97 and AUPRC 0.97 in detecting HFrEF on the test set. In external validation, the model had high performance in identifying HFrEF with AUROC 0.91-0.95 and AUPRC 0.91-0.96 among 19,242 discharge summaries at Northwestern Medicine, 139 notes from Yale community hospitals, and 146 notes from MIMIC-III. Model predictions corresponded to an NRI of 28 ± 7% compared to chart diagnosis (p< 0.001) and identified significant gaps in the use of guideline-directed therapies (Figure). Conclusions: We developed and validated an NLP model that automates the identification of HFrEF from clinical notes with high precision and accuracy, representing a key element to automate quality-of-care measurements in HFrEF.

The effect of conscientiousness on managerial learning from stock prices

ABSTRACT We examine whether and how CEOs’ conscientiousness affects managerial learning from stock prices. We use a deep learning-based natural language processing model to detect personality from textual records of an individual’s spoken language. Following personality psychologists’ description of the personality trait of conscientiousness, we hypothesize that conscientious CEOs are more willing and able to collect comprehensive decision-relevant information compared to others, and to the extent that they possess more relevant information prior to the arrival of news in the stock market, they have less to learn from it. From the sample of large U.S. public firms for which earnings call transcripts are available, we replicate the well-stylized investment-to-price relationship while reporting a weaker relationship for firms with conscientious CEOs. This finding supports the hypothesis that conscientious managers learn less from stock prices.

A Comparative Analysis of Chinese-English Translation Quality Based on ChatGPT: A Case Study of Chinese Characteristic Words

The release of ChatGPT, a large-scale natural language interactive program, at the end of 2022 has once again garnered worldwide attention to artificial intelligence technology. As a deep learning-based natural language processing model, ChatGPT demonstrates unique advantages in the field of translation that are unparalleled by previous machine translation software. Chinese characteristic words, which lack direct equivalents in other cultures or countries, have always posed a challenge in Chinese-English translation, but they are also an essential part of effectively conveying the Chinese story. This article selected some Chinese characteristic words mentioned in the Chinese Government Work Report as translation materials and asked ChatGPT with completing tasks such as translating and polishing the translation according to instructions. By comparing the official Chinese-English translation version and traditional machine translation version, this article analyzed the translation quality of ChatGPT in C-E translation from the perspectives of syntactic structure, coherence, and domain adjustment, with the aim of summarizing machine-assisted translation methods and translation strategies for Chinese characteristic words.

Deep Learning-Based Natural Language Processing to Automate Esophagitis Severity Grading from the Electronic Health Records

Radiotherapy (RT) toxicities can impair survival and quality-of-life, yet their risk factors and optimal management are under-studied. Real-world evidence holds enormous potential to improve our understanding of RT adverse events, but this information is often only documented in clinic notes and cannot, at present, be automatically extracted. To address this unmet need, we developed natural language processing (NLP) algorithms to automatically identify the presence and severity of esophagitis from notes of patients treated with thoracic RT. Our corpus consisted of (1) a gold-labeled dataset of 1524 clinic notes from 124 lung cancer patients treated with RT (in-domain), manually annotated for CTCAE v5.0 esophagitis grade, and (2) a silver-labeled dataset of 2420 notes from 1832 patients on whom toxicity grades had been collected as structured data during clinical care. We developed a fine-tuning pipeline for pre-trained BERT-based neural models for 3 tasks: 1) classifying the presence of esophagitis, 2) classifying grade 0-1 vs. > = 2 esophagitis and 3) classifying grade 0 vs. 1 vs. 2-3. A note sectionizer was used to let the model focus on the most informative sections. Independent validation in a separate clinical cohort of esophageal cancer patients was selected for out-of-domain transferability testing. Such cohorts consist of a manually annotated dataset of 345 notes from 75 esophageal cancer patients treated with RT. We also report patient-level results by evaluating the maximum predicted grade per patient. Fine-tuning PubmedBERT yielded the best-performing models. Performance is shown in the table. Selecting the most informative note sections (primarily Interval History, Assessment & Plan) during fine-tuning improved macro-F1 by > = 2% for all tasks. Including silver-labeled data improved the macro-F1 by > = 3% across all tasks. To the best of our knowledge, this is the first effort to automatically extract toxicity severity according to CTCAE guidelines from clinic notes, providing proof-of-concept for NLP to support detailed toxicity reporting. Fine-tuning on note sections and leveraging silver-labeled data enabled promising performance despite small datasets, informing future research into NLP for automated toxicity monitoring. Future work will extend these methods to other cancer diagnoses and toxicities, and to toxicity risk prediction.

Text Classification of Modern Mongolian Legal Documents Using BERT Models

This paper investigates the application of deep learning-based natural language processing techniques in modern Mongolian legal documents. In particular, we explore several methods that apply Bidirectional Encoder Representations from Transformers (BERT) models for classifying modern Mongolian legal documents. Based on our findings, we propose BERT-based models called LEGAL-BERT-Mongolian. We demonstrated two variants of LEGAL-BERT-Mongolian, i.e., uncased-LEGAL-BERT-Mongolian and cased-LEGAL-BERT-Mongolian, for classifying modern Mongolian legal documents. The uncased-LEGAL-BERT-Mongolian model achieved the best results, with a precision of 0.91, recall of 0.87, and F1 score of 0.89, whereas the cased-LEGAL-BERT-Mongolian model achieved a precision of 0.87, recall of 0.83, and F1 score of 0.85.

Natural language processing to convert unstructured COVID-19 chest-CT reports into structured reports

BackgroundStructured reporting has been demonstrated to increase report completeness and to reduce error rate, also enabling data mining of radiological reports. Still, structured reporting is perceived by radiologists as a fragmented reporting style, limiting their freedom of expression. PurposeA deep learning-based natural language processing method was developed to automatically convert unstructured COVID-19 chest CT reports into structured reports. MethodsTwo hundred-two COVID-19 chest CT were retrospectively reviewed by two experienced radiologists, who wrote for each exam a free-form text radiological report and coherently filled the template provided by the Italian Society of Medical and Interventional Radiology, used as ground-truth. A semi-supervised convolutional neural network was implemented to extract 62 categorical variables from the report. Two iterations were carried-out, the first without fine-tuning, the second one performing a fine-tuning. The performance was measured using the mean accuracy and the F1 mean score. An error analysis was performed to identify errors entirely attributable to incorrect processing of the model. ResultsThe algorithm achieved a mean accuracy of 93.7% and an F1 score 93.8% in the first iteration. Most of the errors were exclusively attributable to wrong inference (46%). In the second iteration the model achieved for both parameters 95,8% and percentage of errors attributable to wrong inference decreased to 26%. ConclusionsThe convolutional neural network achieved an optimal performance in the automated conversion of free-form text into structured radiological reports, overcoming all the limitation attributed to structured reporting and finally paving the way for data mining of radiological report.

Exploring the effect of digital CSR communication on firm performance: A deep learning approach

This study proposes a novel research framework to examine the effect of digital CSR communications on financial performance while incorporating deep learning techniques to identify firms' CSR communications on social media. Particularly, this research aims to quantify firms' efforts in digital CSR communications by employing cutting-edge deep learning-based natural language processing (NLP) models to detect CSR-related tweets on social media. Utilizing a unique dataset of 65 Chinese public companies in the manufacturing sector between 2015 and 2019, we detected 64,769 long-form tweets posted on WeChat to acquire both digital CSR communications and stakeholder engagement data. Combining financial and secondary data of sample firms, this research reveals the positive but time-lagged influence of digital CSR communications on firms' financial performance, primarily through the lens of agenda-setting theory. We also find that stakeholder engagement plays an essential bridging role in the relationship above, while CSR ratings surprisingly hamper such a positive effect.

Deep learning approach to detection of colonoscopic information from unstructured reports

BackgroundColorectal cancer is a leading cause of cancer deaths. Several screening tests, such as colonoscopy, can be used to find polyps or colorectal cancer. Colonoscopy reports are often written in unstructured narrative text. The information embedded in the reports can be used for various purposes, including colorectal cancer risk prediction, follow-up recommendation, and quality measurement. However, the availability and accessibility of unstructured text data are still insufficient despite the large amounts of accumulated data. We aimed to develop and apply deep learning-based natural language processing (NLP) methods to detect colonoscopic information.MethodsThis study applied several deep learning-based NLP models to colonoscopy reports. Approximately 280,668 colonoscopy reports were extracted from the clinical data warehouse of Samsung Medical Center. For 5,000 reports, procedural information and colonoscopic findings were manually annotated with 17 labels. We compared the long short-term memory (LSTM) and BioBERT model to select the one with the best performance for colonoscopy reports, which was the bidirectional LSTM with conditional random fields. Then, we applied pre-trained word embedding using large unlabeled data (280,668 reports) to the selected model.ResultsThe NLP model with pre-trained word embedding performed better for most labels than the model with one-hot encoding. The F1 scores for colonoscopic findings were: 0.9564 for lesions, 0.9722 for locations, 0.9809 for shapes, 0.9720 for colors, 0.9862 for sizes, and 0.9717 for numbers.ConclusionsThis study applied deep learning-based clinical NLP models to extract meaningful information from colonoscopy reports. The method in this study achieved promising results that demonstrate it can be applied to various practical purposes.

A developer-oriented recommender model for the app store: A predictive network analytics approach

While thousands of new mobile applications (i.e., apps) are being added to the major app markets daily, only a small portion of them attain their financial goals and survive in these competitive marketplaces. A key to the quick growth and success of relatively less popular apps is that they should make their way to the limited list of apps recommended to users of already popular apps; however, the focus of the current literature on consumers has created a void of design principles for app developers. In this study, employing a predictive network analytics approach combined with deep learning-based natural language processing and explainable artificial intelligence techniques, we shift the focus from consumers and propose a developer-oriented recommender model. We employ a set of app-specific and network-driven variables to present a novel approach for predicting potential recommendation relationships among apps, which enables app developers and marketers to characterize and target appropriate consumers. We validate the proposed model using a large (>23,000), longitudinal dataset of medical apps collected from the iOS App Store at two time points. From a total of 10,234 network links (recommendations) formed between the two data collection points, the proposed approach was able to correctly predict 8,780 links (i.e., 85.8 %). We perform Shapley Additive exPlanation (SHAP) analysis to identify the most important determinants of link formations and provide insights for the app developers about the factors and design principles they can incorporate into their development process to maximize the chances of success for their apps.

Getting More Out of Large Databases and EHRs with Natural Language Processing and Artificial Intelligence: The Future Is Here.

Electronic health records (EHRs) have created great opportunities to collect various information from clinical patient encounters. However, most EHR data are stored in unstructured form (e.g., clinical notes, surgical notes, and medication instructions), and researchers need data to be in computable form (structured) to extract meaningful relationships involving variables that can influence patient outcomes. Clinical natural language processing (NLP) is the field of extracting structured data from unstructured text documents in EHRs. Clinical text has several characteristics that mandate the use of special techniques to extract structured information from them compared with generic NLP methods. In this article, we define clinical NLP models, introduce different methods of information extraction from unstructured data using NLP, and describe the basic technical aspects of how deep learning-based NLP models work. We conclude by noting the challenges of working with clinical NLP models and summarizing the general steps needed to launch an NLP project.

From Text Representation to Financial Market Prediction: A Literature Review

News dissemination in social media causes fluctuations in financial markets. (Scope) Recent advanced methods in deep learning-based natural language processing have shown promising results in financial market analysis. However, understanding how to leverage large amounts of textual data alongside financial market information is important for the investors’ behavior analysis. In this study, we review over 150 publications in the field of behavioral finance that jointly investigated natural language processing (NLP) approaches and a market data analysis for financial decision support. This work differs from other reviews by focusing on applied publications in computer science and artificial intelligence that contributed to a heterogeneous information fusion for the investors’ behavior analysis. (Goal) We study various text representation methods, sentiment analysis, and information retrieval methods from heterogeneous data sources. (Findings) We present current and future research directions in text mining and deep learning for correlation analysis, forecasting, and recommendation systems in financial markets, such as stocks, cryptocurrencies, and Forex (Foreign Exchange Market).

Predicting the sentiment of South Korean Twitter users toward vaccination after the emergence of COVID-19 Omicron variant using deep learning-based natural language processing.

Although the full vaccination rate of South Korea compared to other countries, concerns about the effectiveness of the vaccine are growing as new COVID variants such as Alpha, Beta, Gamma, Delta, and Omicron appear over time. In this study, we collected Twitter data in South Korea that contained keywords like vaccines after the outbreak of the Omicron variant from 27 November 2021 to 14 February 2022. First, we analyzed the relationship between potential keywords associated with vaccination after the appearance of the Omicron variant in Twitter using network analysis. Second, we developed an efficient model for predicting the emotion of speech regarding vaccination after the COVID-19 Omicron variant pandemic by using deep learning algorithms. We constructed sentiment analysis models regarding vaccination after the COVID-19 Omicron pandemic by using five algorithms [i.e., support vector machine (SVM), recurrent neural networks (RNNs), long short-term memory models (LSTMs), bidirectional encoder representations from transformers (BERT), and Korean BERT (KoBERT)]. The results confirmed that KoBERT showed the best performance (71%) in all predictive performance indicators (accuracy, precision, and F1 score). It is necessary to prepare measures to alleviate the negative factorss of the public about vaccination in the long-term pandemic situation and help the public recognize the efficacy and safety of vaccination by using big data based on the results of this study.

“Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks

One unintended consequence of the Electronic Health Records (EHR) implementation is the overuse of content-importing technology, such as copy-and-paste, that creates “bloated” notes containing large amounts of textual redundancy. Despite the rising interest in applying machine learning models to learn from real-patient data, it is unclear how the phenomenon of note bloat might affect the Natural Language Processing (NLP) models derived from these notes. Therefore, in this work we examine the impact of redundancy on deep learning-based NLP models, considering four clinical prediction tasks using a publicly available EHR database. We applied two deduplication methods to the hospital notes, identifying large quantities of redundancy, and found that removing the redundancy usually has little negative impact on downstream performances, and can in certain circumstances assist models to achieve significantly better results. We also showed it is possible to attack model predictions by simply adding note duplicates, causing changes of correct predictions made by trained models into wrong predictions. In conclusion, we demonstrated that EHR text redundancy substantively affects NLP models for clinical prediction tasks, showing that the awareness of clinical contexts and robust modeling methods are important to create effective and reliable NLP systems in healthcare contexts.

Deep Learning-Based Natural Language Processing Methods for Sentiment Analysis in Social Networks

This paper presents an in-depth study of the sentiment of social network communication through a deep learning-based natural language processing approach and designs a corresponding model to be applied in the actual social process. Specifically, the network can dynamically select the most important word in the current state according to the information available and achieve the accurate recognition of the dynamically changing important content in a sentence. Based on this, the semantic understanding of the whole sentence is achieved through a continuous cycle of the process. In addition, considering that the semantic representation of natural language is highly dependent on contextual information, the lack of contextual information will lead to the ambiguity and inaccuracy of semantic representation. In this paper, we study the sentiment analysis algorithms in social networks at two levels, unimodal and multimodal, and construct a text sentiment analysis model and a picture-text multimodal sentiment analysis model in social networks, respectively. By comparing the experiments with the existing models on several datasets, the accuracy of the two models exceeded the benchmark models by 4.45% and 5.2%, respectively, which verified the effectiveness of the two models. The feasibility of applying the optimized convolutional neural network recurrent optimization network to social network sentiment analysis is verified by practically applying the optimized convolutional neural network recurrent optimization network to single task and multitask and comparing other existing deep learning classifiers.

Using deep learning-based natural language processing to identify reasons for statin nonuse in patients with atherosclerotic cardiovascular disease

BackgroundStatins conclusively decrease mortality in atherosclerotic cardiovascular disease (ASCVD), the leading cause of death worldwide, and are strongly recommended by guidelines. However, real-world statin utilization and persistence are low, resulting in excess mortality. Identifying reasons for statin nonuse at scale across health systems is crucial to developing targeted interventions to improve statin use.MethodsWe developed and validated deep learning-based natural language processing (NLP) approaches (Clinical Bidirectional Encoder Representations from Transformers [BERT]) to classify statin nonuse and reasons for statin nonuse using unstructured electronic health records (EHRs) from a diverse healthcare system.ResultsWe present data from a cohort of 56,530 ASCVD patients, among whom 21,508 (38%) lack guideline-directed statin prescriptions and statins listed as allergies in structured EHR portions. Of these 21,508 patients without prescriptions, only 3,929 (18%) have any discussion of statin use or nonuse in EHR documentation. The NLP classifiers identify statin nonuse with an area under the curve (AUC) of 0.94 (95% CI 0.93–0.96) and reasons for nonuse with a weighted-average AUC of 0.88 (95% CI 0.86–0.91) when evaluated against manual expert chart review in a held-out test set. Clinical BERT identifies key patient-level reasons (side-effects, patient preference) and clinician-level reasons (guideline-discordant practices) for statin nonuse, including differences by type of ASCVD and patient race/ethnicity.ConclusionsOur deep learning NLP classifiers can identify crucial gaps in statin nonuse and reasons for nonuse in high-risk populations to support education, clinical decision support, and potential pathways for health systems to address ASCVD treatment gaps.