Deep Learning-based Natural Language Processing Research Articles

Supporting the care to breast cancer patients with unique needs: Evidence from online community members’ responses

BackgroundBreast cancer is the most common cancer diagnosed in women globally. Online cancer communities (OCCs) provide platforms for breast cancer patients to connect, share experiences, and support each other. These communities facilitate discussions on a range of health- and non-health-related topics. However, posts discussing unique topics may receive varying levels of attention and support. This study aims to devise a method for identifying and supporting such posts, enhancing community response and support strategies. MethodsWe propose a Uniqueness Score Extraction Framework to compute health- and non-health-related uniqueness scores for online community posts. The framework utilizes deep learning-based natural language processing models to identify the topics discussed in OCCs and calculates the health- and non-health-related uniqueness scores of a post based on the uniqueness of the topics identified by the BERTopic model. We further employ econometric models to assess how the uniqueness scores of posts affect community members’ responses to those posts. ResultsOur study reveals that posts with a higher concentration of unique health-related topics in OCCs elicit quicker, more frequent, but shorter responses. Conversely, posts containing more unique non-health-related topics in the entire post prompt faster and longer responses, unless these topics become overly dominant, in which case the number of replies decreases, and response times are prolonged. ConclusionOur research develops a framework to identify posts with high uniqueness scores in OCCs, and sheds light on community member responses to these discussions. The findings indicate that while members are supportive, particularly regarding health-related topics, the post-content’s nature and focus greatly affect their engagement. These discoveries could enhance our understanding of community dynamics in OCCs, offering valuable implications for researchers, OCC facilitators, and medical professionals in supporting patients within online platforms.

Automated Identification of HeartFailureWith Reduced Ejection Fraction Using Deep Learning-Based Natural Language Processing.

The lack of automated tools for measuring care quality limits the implementation of a national program to assess guideline-directed care in heart failure with reduced ejection fraction (HFrEF). The authors aimed to automate the identification of patients with HFrEF at hospital discharge, an opportunity to evaluate and improve the quality of care. The authors developed a novel deep-learning language model for identifying patients with HFrEF from discharge summaries of hospitalizations with heart failure at Yale New Haven Hospital during 2015 to 2019. HFrEF was defined by left ventricular ejection fraction<40% on antecedent echocardiography. The authors externally validated the model at Northwestern Medicine, community hospitals of Yale, and the MIMIC-III (Medical Information Mart for Intensive Care III) database. A total of 13,251 notes from 5,392 unique individuals (age 73 ± 14 years, 48% women), including 2,487 patients with HFrEF (46.1%), were used for model development (train/held-out: 70%/30%). The model achieved an area under receiver-operating characteristic curve (AUROC) of 0.97 and area under precision recall curve (AUPRC) of 0.97 in detecting HFrEF on the held-out set. The model had high performance in identifying HFrEF with AUROC=0.94 and AUPRC=0.91 on 19,242 notes from Northwestern Medicine, AUROC=0.95 and AUPRC=0.96 on 139 manually abstracted notes from Yale community hospitals, and AUROC=0.91 and AUPRC=0.92 on 146 manually reviewed notes from MIMIC-III. Model-based predictions of HFrEF corresponded to a net reclassification improvement of 60.2 ± 1.9% compared with diagnosis codes (P< 0.001). The authors developed a language model that identifies HFrEF from clinical notes with high precision and accuracy, representing a key element in automating quality assessment for individuals with HFrEF.

NLP-based ergonomics MSD risk root cause analysis and risk controls recommendation

An ergonomics assessment of the physical risk factors in the workplace is instrumental in predicting and preventing musculoskeletal disorders (MSDs). Using Artificial Intelligence (AI) has become increasingly popular for ergonomics assessments because of the time savings and improved accuracy. However, most of the effort in this area starts and ends with producing risk scores, without providing guidance to reduce the risk. This paper proposes a holistic job improvement process that performs automatic root cause analysis and control recommendations for reducing MSD risk. We apply deep learning-based Natural Language Processing (NLP) techniques such as Part of Speech (PoS) tagging and dependency parsing on textual descriptions of the physical actions performed in the job (e.g. pushing) along with the object (e.g. cart) being acted upon. The action-object inferences provide the entry point to an expert-based Machine Learning (ML) system that automatically identifies the targeted work-related causes (e.g. cart movement forces are too high, due to caster size too small) of the identified MSD risk (e.g. excessive shoulder forces). The proposed framework utilises the root causes identified to recommend control strategies (e.g. provide larger diameter casters, minimum diameter 8" or 203 mm) most likely to mitigate risk, resulting in a more efficient and effective job improvement process.

Manager sentiment, policy uncertainty, ESG disclosure and firm performance: a large language model in corporate landscape

Purpose This study aims to examine the impact of manager sentiment on the firm performance (FP) of Indian-listed nonfinancial firms. Further, it endeavors to investigate the moderating role of economic policy uncertainty (EPU) and environment, social and governance (ESG) transparency in this relationship. Design/methodology/approach A noble manager sentiment is introduced using FinBERT, a bidirectional encoder representation from a transformers (BERT)-type large language model. Using this deep learning-based natural language processing approach implemented through a Python-generated algorithm, this study constructs a manager sentiment for each firm and year based on the management discussions and analysis (MD&amp;A) report. This research uses the system GMM to examine how manager sentiment affects FP. Findings The empirical results suggest that managers’ optimistic outlook in MD&amp;A corporate disclosure sections tends to present higher performance. This positive association remains consistent after several robustness checks – using propensity score matching and instrumental variable approach to address further endogeneity, using alternative proxies of manager sentiment and FP and conducting subsample analysis based on financial constraints. Furthermore, the authors observe that the relationship is more pronounced for ESG-disclosed firms and during the low EPU. Practical implications The results demonstrate that the manager sentiment strongly predicts FP. Thus, this study may provide valuable insight for academics, practitioners, investors, corporates and policymakers. Originality/value To the best of the authors’ knowledge, this is the first study to predict FP by using FinBERT-based managerial sentiment, particularly in an emerging market context.

Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing.

The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7-47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.

How to strategically respond to online hotel reviews: A strategy-aware deep learning approach

Online reviews exert a considerable influence on consumer purchase behavior, yet there remains ambiguity about the most effective managerial response strategies for positive and negative reviews. Addressing this gap, our study introduces a Strategy-Aware, Deep Learning-Based Natural Language Processing (Sa-DLNLP) model designed to optimize firm responses. The proposed model underwent rigorous evaluation through a human-coded study and was subsequently validated by a separate user response study. Our findings reveal that active-constructive responses significantly enhance the impact of positive reviews, whereas passive-constructive strategies are more effective in mitigating the damage from negative reviews. Additionally, the study underscores the importance of concise, personalized, and prompt responses across the board. Interestingly, responses that are overly explanatory, excessively empathetic, or challenge customers were found to be counterproductive when dealing with negative reviews. This study not only demystifies the art of managing online reviews but also offers an advanced deep learning methodology that can directly benefit the disciplines of Information Systems and Management.

Optimisation Study of Natural Language Processing Algorithms Based on Deep Learning

This paper presents a comprehensive optimization study of natural language processing (NLP) algorithms based on deep learning techniques. The research explores various strategies to enhance the performance and efficiency of NLP models, aiming to address the challenges posed by large-scale datasets and complex linguistic structures. Through a systematic review of existing literature and methodologies, this study synthesizes insights into key optimization approaches, including model architecture design, parameter tuning, and data preprocessing techniques. Moreover, it investigates the impact of different optimization strategies on NLP tasks such as sentiment analysis, named entity recognition, and machine translation. By elucidating the strengths and limitations of various optimization techniques, this paper offers valuable insights for researchers and practitioners in the field of deep learning-based NLP.

Application of Deep Learning-Based Natural Language Processing in Multilingual Sentiment Analysis

This study explores the application of deep learning-based natural language processing technologies in multilingual sentiment analysis. By examining the performance of deep learning models such as BERT and LSTM in multilingual settings, the research demonstrates the effectiveness of these models in cross-linguistic sentiment classification tasks. Despite progress, major challenges in multilingual sentiment analysis include language and cultural differences, insufficient complex context processing, and data imbalance. Future research directions include optimizing the models' contextual understanding abilities, leveraging multilingual data resources, exploring novel neural network architectures, and improving assessment metrics. With these measures, the accuracy and efficiency of multilingual sentiment analysis are expected to be significantly enhanced, further advancing the global application of natural language processing technologies.

Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triage

ObjectiveThe manual recording of electronic health records (EHRs) by clinicians in the emergency department (ED) is time-consuming and challenging. In light of recent advancements in large language models (LLMs) such as GPT and BERT, this study aimed to design and validate LLMs for automatic clinical diagnoses. The models were designed to identify 12 medical symptoms and 2 patient histories from simulated clinician–patient conversations within 6 primary symptom scenarios in emergency triage rooms. Materials and methodWe developed classification models by fine-tuning BERT, a transformer-based pre-trained model. We subsequently analyzed these models using eXplainable artificial intelligence (XAI) and the Shapley additive explanation (SHAP) method. A Turing test was conducted to ascertain the reliability of the XAI results by comparing them to the outcomes of tasks performed and explained by medical workers. An emergency medicine specialist assessed the results of both XAI and the medical workers. ResultsWe fine-tuned four pre-trained LLMs and compared their classification performance. The KLUE-RoBERTa-based model demonstrated the highest performance (F1-score: 0.965, AUROC: 0.893) on human-transcribed script data. The XAI results using SHAP showed an average Jaccard similarity of 0.722 when compared with explanations of medical workers for 15 samples. The Turing test results revealed a small 6% gap, with XAI and medical workers receiving the mean scores of 3.327 and 3.52, respectively. ConclusionThis paper highlights the potential of LLMs for automatic EHR recording in Korean EDs. The KLUE-RoBERTa-based model demonstrated superior classification performance. Furthermore, XAI using SHAP provided reliable explanations for model outputs. The reliability of these explanations was confirmed by a Turing test.

미디어 텍스트 분석 기반의 공급망 리스크 모니터링 시스템의 개발

Recently, companies are exposed to various supply chain risks such as intensified trade conflicts, epidemics, economic and geopolitical uncertainties, and natural disasters. Thus there is increasing importance in monitoring information related to supply chain risks. Analyzing real-time media texts, such as news articles, can be utilized for monitoring up-to-date information supply chain risks. However, researches regarding analyzing supply chain risk related text are in early stages, and researches to apply modern AI techniques such as deep learning-based natural language processing to supply chain risk texts are scarce. This study aims to develop a supply chain risk monitoring system that monitors and extracts information related to supply chain risks by analyzing news articles. To collect supply chain risk related articles a filtering model based on KoBERT is developed, of which risk types are identified based on LDA topic modeling to be utilized as the train data. To predict news articles’ risk types, two deep learning- based risk classification models are developed using BOW(Bag of Words) and KoBERT. The results showed high accuracy of KoBERT based model in filtering supply chain risk-related articles, and in the classification of supply chain risk types also KoBERT based model showed better performance than BOW based model.

IDPpub: Illuminating the Dark Phosphoproteome Through PubMed Mining

Global phosphoproteomics experiments quantify tens of thousands of phosphorylation sites. However, data interpretation is hampered by our limited knowledge on functions, biological contexts, or precipitating enzymes of the phosphosites. This study establishes a repository of phosphosites with associated evidence in biomedical abstracts, using deep learning-based natural language processing techniques. Our model for illuminating the dark phosphoproteome through PubMed mining (IDPpub) was generated by fine-tuning BioBERT, a deep learning tool for biomedical text mining. Trained using sentences containing protein substrates and phosphorylation site positions from 3000 abstracts, the IDPpub model was then used to extract phosphorylation sites from all MEDLINE abstracts. The extracted proteins were normalized to gene symbols using the National Center for Biotechnology Information gene query, and sites were mapped to human UniProt sequences using ProtMapper and mouse UniProt sequences by direct match. Precision and recall were calculated using 150 curated abstracts, and utility was assessed by analyzing the CPTAC (Clinical Proteomics Tumor Analysis Consortium) pan-cancer phosphoproteomics datasets and the PhosphoSitePlus database. Using 10-fold cross validation, pairs of correct substrates and phosphosite positions were extracted with an average precision of 0.93 and recall of 0.94. After entity normalization and site mapping to human reference sequences, an independent validation achieved a precision of 0.91 and recall of 0.77. The IDPpub repository contains 18,458 unique human phosphorylation sites with evidence sentences from 58,227 abstracts and 5918 mouse sites in 14,610 abstracts. This included evidence sentences for 1803 sites identified in CPTAC studies that are not covered by manually curated functional information in PhosphoSitePlus. Evaluation results demonstrate the potential of IDPpub as an effective biomedical text mining tool for collecting phosphosites. Moreover, the repository (http://idppub.ptmax.org), which can be automatically updated, can serve as a powerful complement to existing resources.

Abstract 16207: Deep Learning-Based Natural Language Processing of Discharge Summaries for Automated Identification of Heart Failure With Reduced Ejection Fraction

Background: Implementation of a national program to improve guideline-directed care in heart failure with reduced ejection fraction (HFrEF) has been limited by the lack of automated tools to measure care quality. A key challenge for this approach is to identify patients with HFrEF at hospital discharge, an opportunity to evaluate and improve care. Methods: We developed a novel deep-learning language model to detect patients with HFrEF from discharge summaries using a semi-supervised learning framework. Hospitalizations with heart failure (HF) at Yale New Haven Hospital between 2015-2019 were labeled as HFrEF if left ventricular ejection fraction was under 40% on antecedent echocardiography. The model was internally validated with model-based net reclassification improvement (NRI) assessed against chart-based diagnosis codes. We externally validated the model on discharge summaries from hospitalizations with HF at Northwestern Medicine, community hospitals of Yale Health, and MIMIC-III database, confirmed with chart abstraction. Results: A total of 13,251 notes from 5,392 individuals (age 73 ± 14 years, 48% female), including 2,487 (46.1%) patients with HFrEF were used for model development (train/validation/test: 70/15/15%). The model achieved an AUROC 0.97 and AUPRC 0.97 in detecting HFrEF on the test set. In external validation, the model had high performance in identifying HFrEF with AUROC 0.91-0.95 and AUPRC 0.91-0.96 among 19,242 discharge summaries at Northwestern Medicine, 139 notes from Yale community hospitals, and 146 notes from MIMIC-III. Model predictions corresponded to an NRI of 28 ± 7% compared to chart diagnosis (p&lt; 0.001) and identified significant gaps in the use of guideline-directed therapies (Figure). Conclusions: We developed and validated an NLP model that automates the identification of HFrEF from clinical notes with high precision and accuracy, representing a key element to automate quality-of-care measurements in HFrEF.

The effect of conscientiousness on managerial learning from stock prices

ABSTRACT We examine whether and how CEOs’ conscientiousness affects managerial learning from stock prices. We use a deep learning-based natural language processing model to detect personality from textual records of an individual’s spoken language. Following personality psychologists’ description of the personality trait of conscientiousness, we hypothesize that conscientious CEOs are more willing and able to collect comprehensive decision-relevant information compared to others, and to the extent that they possess more relevant information prior to the arrival of news in the stock market, they have less to learn from it. From the sample of large U.S. public firms for which earnings call transcripts are available, we replicate the well-stylized investment-to-price relationship while reporting a weaker relationship for firms with conscientious CEOs. This finding supports the hypothesis that conscientious managers learn less from stock prices.

A Comparative Analysis of Chinese-English Translation Quality Based on ChatGPT: A Case Study of Chinese Characteristic Words

The release of ChatGPT, a large-scale natural language interactive program, at the end of 2022 has once again garnered worldwide attention to artificial intelligence technology. As a deep learning-based natural language processing model, ChatGPT demonstrates unique advantages in the field of translation that are unparalleled by previous machine translation software. Chinese characteristic words, which lack direct equivalents in other cultures or countries, have always posed a challenge in Chinese-English translation, but they are also an essential part of effectively conveying the Chinese story. This article selected some Chinese characteristic words mentioned in the Chinese Government Work Report as translation materials and asked ChatGPT with completing tasks such as translating and polishing the translation according to instructions. By comparing the official Chinese-English translation version and traditional machine translation version, this article analyzed the translation quality of ChatGPT in C-E translation from the perspectives of syntactic structure, coherence, and domain adjustment, with the aim of summarizing machine-assisted translation methods and translation strategies for Chinese characteristic words.

Deep Learning-Based Natural Language Processing to Automate Esophagitis Severity Grading from the Electronic Health Records

Radiotherapy (RT) toxicities can impair survival and quality-of-life, yet their risk factors and optimal management are under-studied. Real-world evidence holds enormous potential to improve our understanding of RT adverse events, but this information is often only documented in clinic notes and cannot, at present, be automatically extracted. To address this unmet need, we developed natural language processing (NLP) algorithms to automatically identify the presence and severity of esophagitis from notes of patients treated with thoracic RT. Our corpus consisted of (1) a gold-labeled dataset of 1524 clinic notes from 124 lung cancer patients treated with RT (in-domain), manually annotated for CTCAE v5.0 esophagitis grade, and (2) a silver-labeled dataset of 2420 notes from 1832 patients on whom toxicity grades had been collected as structured data during clinical care. We developed a fine-tuning pipeline for pre-trained BERT-based neural models for 3 tasks: 1) classifying the presence of esophagitis, 2) classifying grade 0-1 vs. > = 2 esophagitis and 3) classifying grade 0 vs. 1 vs. 2-3. A note sectionizer was used to let the model focus on the most informative sections. Independent validation in a separate clinical cohort of esophageal cancer patients was selected for out-of-domain transferability testing. Such cohorts consist of a manually annotated dataset of 345 notes from 75 esophageal cancer patients treated with RT. We also report patient-level results by evaluating the maximum predicted grade per patient. Fine-tuning PubmedBERT yielded the best-performing models. Performance is shown in the table. Selecting the most informative note sections (primarily Interval History, Assessment & Plan) during fine-tuning improved macro-F1 by > = 2% for all tasks. Including silver-labeled data improved the macro-F1 by > = 3% across all tasks. To the best of our knowledge, this is the first effort to automatically extract toxicity severity according to CTCAE guidelines from clinic notes, providing proof-of-concept for NLP to support detailed toxicity reporting. Fine-tuning on note sections and leveraging silver-labeled data enabled promising performance despite small datasets, informing future research into NLP for automated toxicity monitoring. Future work will extend these methods to other cancer diagnoses and toxicities, and to toxicity risk prediction.

Text Classification of Modern Mongolian Legal Documents Using BERT Models

This paper investigates the application of deep learning-based natural language processing techniques in modern Mongolian legal documents. In particular, we explore several methods that apply Bidirectional Encoder Representations from Transformers (BERT) models for classifying modern Mongolian legal documents. Based on our findings, we propose BERT-based models called LEGAL-BERT-Mongolian. We demonstrated two variants of LEGAL-BERT-Mongolian, i.e., uncased-LEGAL-BERT-Mongolian and cased-LEGAL-BERT-Mongolian, for classifying modern Mongolian legal documents. The uncased-LEGAL-BERT-Mongolian model achieved the best results, with a precision of 0.91, recall of 0.87, and F1 score of 0.89, whereas the cased-LEGAL-BERT-Mongolian model achieved a precision of 0.87, recall of 0.83, and F1 score of 0.85.

Natural language processing to convert unstructured COVID-19 chest-CT reports into structured reports

BackgroundStructured reporting has been demonstrated to increase report completeness and to reduce error rate, also enabling data mining of radiological reports. Still, structured reporting is perceived by radiologists as a fragmented reporting style, limiting their freedom of expression. PurposeA deep learning-based natural language processing method was developed to automatically convert unstructured COVID-19 chest CT reports into structured reports. MethodsTwo hundred-two COVID-19 chest CT were retrospectively reviewed by two experienced radiologists, who wrote for each exam a free-form text radiological report and coherently filled the template provided by the Italian Society of Medical and Interventional Radiology, used as ground-truth. A semi-supervised convolutional neural network was implemented to extract 62 categorical variables from the report. Two iterations were carried-out, the first without fine-tuning, the second one performing a fine-tuning. The performance was measured using the mean accuracy and the F1 mean score. An error analysis was performed to identify errors entirely attributable to incorrect processing of the model. ResultsThe algorithm achieved a mean accuracy of 93.7% and an F1 score 93.8% in the first iteration. Most of the errors were exclusively attributable to wrong inference (46%). In the second iteration the model achieved for both parameters 95,8% and percentage of errors attributable to wrong inference decreased to 26%. ConclusionsThe convolutional neural network achieved an optimal performance in the automated conversion of free-form text into structured radiological reports, overcoming all the limitation attributed to structured reporting and finally paving the way for data mining of radiological report.

Exploring the effect of digital CSR communication on firm performance: A deep learning approach

This study proposes a novel research framework to examine the effect of digital CSR communications on financial performance while incorporating deep learning techniques to identify firms' CSR communications on social media. Particularly, this research aims to quantify firms' efforts in digital CSR communications by employing cutting-edge deep learning-based natural language processing (NLP) models to detect CSR-related tweets on social media. Utilizing a unique dataset of 65 Chinese public companies in the manufacturing sector between 2015 and 2019, we detected 64,769 long-form tweets posted on WeChat to acquire both digital CSR communications and stakeholder engagement data. Combining financial and secondary data of sample firms, this research reveals the positive but time-lagged influence of digital CSR communications on firms' financial performance, primarily through the lens of agenda-setting theory. We also find that stakeholder engagement plays an essential bridging role in the relationship above, while CSR ratings surprisingly hamper such a positive effect.

Optimizing Clinical Trial Eligibility Design Using Natural Language Processing Models and Real-World Data: Algorithm Development and Validation.

Clinical trials are vital for developing new therapies but can also delay drug development. Efficient trial data management, optimized trial protocol, and accurate patient identification are critical for reducing trial timelines. Natural language processing (NLP) has the potential to achieve these objectives. This study aims to assess the feasibility of using data-driven approaches to optimize clinical trial protocol design and identify eligible patients. This involves creating a comprehensive eligibility criteria knowledge base integrated within electronic health records using deep learning-based NLP techniques. We obtained data of 3281 industry-sponsored phase 2 or 3 interventional clinical trials recruiting patients with non-small cell lung cancer, prostate cancer, breast cancer, multiple myeloma, ulcerative colitis, and Crohn disease from ClinicalTrials.gov, spanning the period between 2013 and 2020. A customized bidirectional long short-term memory- and conditional random field-based NLP pipeline was used to extract all eligibility criteria attributes and convert hypernym concepts into computable hyponyms along with their corresponding values. To illustrate the simulation of clinical trial design for optimization purposes, we selected a subset of patients with non-small cell lung cancer (n=2775), curated from the Mount Sinai Health System, as a pilot study. We manually annotated the clinical trial eligibility corpus (485/3281, 14.78% trials) and constructed an eligibility criteria-specific ontology. Our customized NLP pipeline, developed based on the eligibility criteria-specific ontology that we created through manual annotation, achieved high precision (0.91, range 0.67-1.00) and recall (0.79, range 0.50-1) scores, as well as a high F1-score (0.83, range 0.67-1), enabling the efficient extraction of granular criteria entities and relevant attributes from 3281 clinical trials. A standardized eligibility criteria knowledge base, compatible with electronic health records, was developed by transforming hypernym concepts into machine-interpretable hyponyms along with their corresponding values. In addition, an interface prototype demonstrated the practicality of leveraging real-world data for optimizing clinical trial protocols and identifying eligible patients. Our customized NLP pipeline successfully generated a standardized eligibility criteria knowledge base by transforming hypernym criteria into machine-readable hyponyms along with their corresponding values. A prototype interface integrating real-world patient information allows us to assess the impact of each eligibility criterion on the number of patients eligible for the trial. Leveraging NLP and real-world data in a data-driven approach holds promise for streamlining the overall clinical trial process, optimizing processes, and improving efficiency in patient identification.

Natural Language Processing for Clinical Laboratory Data Repository Systems: Implementation and Evaluation for Respiratory Viruses.

With the growing volume and complexity of laboratory repositories, it has become tedious to parse unstructured data into structured and tabulated formats for secondary uses such as decision support, quality assurance, and outcome analysis. However, advances in natural language processing (NLP) approaches have enabled efficient and automated extraction of clinically meaningful medical concepts from unstructured reports. In this study, we aimed to determine the feasibility of using the NLP model for information extraction as an alternative approach to a time-consuming and operationally resource-intensive handcrafted rule-based tool. Therefore, we sought to develop and evaluate a deep learning-based NLP model to derive knowledge and extract information from text-based laboratory reports sourced from a provincial laboratory repository system. The NLP model, a hierarchical multilabel classifier, was trained on a corpus of laboratory reports covering testing for 14 different respiratory viruses and viral subtypes. The corpus includes 87,500 unique laboratory reports annotated by 8 subject matter experts (SMEs). The classification task involved assigning the laboratory reports to labels at 2 levels: 24 fine-grained labels in level 1 and 6 coarse-grained labels in level 2. A "label" also refers to the status of a specific virus or strain being tested or detected (eg, influenza A is detected). The model's performance stability and variation were analyzed across all labels in the classification task. Additionally, the model's generalizability was evaluated internally and externally on various test sets. Overall, the NLP model performed well on internal, out-of-time (pre-COVID-19), and external (different laboratories) test sets with microaveraged F1-scores >94% across all classes. Higher precision and recall scores with less variability were observed for the internal and pre-COVID-19 test sets. As expected, the model's performance varied across categories and virus types due to the imbalanced nature of the corpus and sample sizes per class. There were intrinsically fewer classes of viruses being detected than those tested; therefore, the model's performance (lowest F1-score of 57%) was noticeably lower in the detected cases. We demonstrated that deep learning-based NLP models are promising solutions for information extraction from text-based laboratory reports. These approaches enable scalable, timely, and practical access to high-quality and encoded laboratory data if integrated into laboratory information system repositories.