Natural Language Processing Pipeline Research Articles

Enhancing Aortic Aneurysm Surveillance: Transformer Natural Language Processing for Flagging and Measuring in Radiology Reports

Incidental findings of aortic aneurysms (AAs) often go unreported, and established patients are frequently lost to follow-up. Natural language processing (NLP) offers a promising solution to address these issues. While rule-based NLP methods have shown some success, recent advancements in transformer-based large language models (LLMs) remain underutilized. This study has three aims: (1) to evaluate the effectiveness of our innovative transformer-based NLP pipeline regarding AA detection; (2) to detail the clinical impact by quantifying the number of patients who could benefit from such technology; and (3) to use this information to help coordinate appointments with patients, ensuring proper monitoring and management. 3229 radiology reports were divided into three batches with varying class balance. Each entry was processed through our innovative NLP pipeline, where it was fragmented using regular expression (regex) functions to isolate relevant textual segments. These segments were subsequently processed through our "question and find" (Q&F) function, powered by Google's BERT, a well-established transformer LLM. This Q&F function extracted aortic diameter measurements, flagging measurements that exceeded a predefined threshold. Following detection, we conducted comprehensive chart reviews and contacted primary care providers (PCPs) and patients to categorize aneurysms as "known" or "incidental." We also assessed whether patients with known aneurysms were adhering to regular yearly screenings and coordinated follow-up appointments. Evaluation of the three batches showed high F1 scores: 99.4% (95% CI [98.5-100]), 96.7% (95% CI [95.0-98.2]), and 98.9% (95% CI [98.0-99.6]). Overall measurement accuracy was 98.9% (95% CI [97.6-100]), 99.6% (95% CI [99.3-99.9]), and 98.1% (95% CI [96.8-99.4]). Compared to manual chart reviews, the NLP system demonstrated superior accuracy and fewer errors: 12 vs. 22 (p=0.084), 47 vs. 98 (p=0.000021), and 31 vs. 53 (p=0.015). Of the 412 patients investigated, 58 (14.1%) involved incidental findings, 54 patients (15.3%) were lost to follow-up, 39 patients (55.7%) were successfully contacted, and 37 follow-up appointments (12.1%) were successfully coordinated. The high-performance metrics from our study demonstrate that transformer-based NLP can enhance aortic aneurysm surveillance. Our subsequent comprehensive patient profiling highlighted the need for such a system as a safety net within the electronic medical record (EMR), systematically reviewing radiology reports to detect incidental findings and patients lost to follow-up. This ensures appropriate referrals and monitoring, improving patient outcomes and healthcare efficiency through timely clinical interventions.

Using Artificial Intelligence to Support Informed Decision-Making on BRAF Mutation Testing.

Precision oncology relies on accurate and interpretable reporting of testing and mutation rates. Focusing on the BRAFV600 mutations in advanced colorectal carcinoma, non-small-cell lung carcinoma, and cutaneous melanoma, we developed a platform displaying testing and mutation rates reported in the literature, which we annotated using an artificial intelligence (AI) and natural language processing (NLP) pipeline. Using AI, we identified publications that likely reported a testing or mutation rate, filtered publications for cancer type, and identified sentences that likely reported rates. Rates and covariates were subsequently manually curated by three experts. The AI performance was evaluated using precision and recall metrics. We used an interactive platform to explore and present the annotated testing and mutation rates by certain study characteristics. The interactive dashboard, accessible at the BRAF dimensions website, enables users to filter mutation and testing rates with relevant options (eg, country of study, study type, mutation type) and to visualize annotated rates. The AI pipeline demonstrated excellent filtering performance (>90% precision and recall for all target cancer types) and moderate performance for sentence classification (53%-99% precision; ≥75% recall). The manual annotation of testing and mutation rates revealed inter-rater disagreement (testing rate, 19%; mutation rate, 70%), indicating unclear or nonstandard reporting of rates in some publications. Our AI-driven NLP pipeline demonstrated the potential for annotating biomarker testing and mutation rates. The difficulties we encountered highlight the need for more advanced AI-powered literature searching and data extraction, and more consistent reporting of testing rates. These improvements would reduce the risk of misinterpretation or misunderstanding of testing and mutation rates by AI-based technologies and the health care community, with beneficial impacts on clinical decision-making, research, and trial design.

Improving surveillance of emergency department activity through natural language processing

Objective and ApproachManagers, clinicians and policymakers have limited information on near-real time patterns of attendance at emergency departments (ED). Our aim was to develop a surveillance system through extracting information from the presenting complaint to develop and feed activity dashboards, improve injury surveillance and support research. We developed a Natural Language Processing (NLP) pipeline using the General Architecture for Text Engineering (GATE) system. The pipeline produced Unified Medical Language System (UMLS) codes from ED presenting complaint records and also implemented the Joint Action on Monitoring Injuries in Europe (JAMIE) Minimum Dataset. We classified complaints into 31 common problem presentations (Coughs, Bleeds, Mental Health, Dizziness...), along with categories from JAMIE (Mechanism of Injury, Location, Activity, Intent…). ResultsPreliminary results are from a pilot study of 86,416 presenting complaints from Morriston Hospital in 2014. The most common terms used for grouped symptoms/categories were “Injury” (38.6%, n=28691), “Pain” (22.8%, n=19420), “Unwell” (7.2%, n=6117), “Bleeding” (6.7%, n=5760), and “Shortness of Breath” (3.8%, n=3255). 32.3% of records contained Mechanism of Injury codes, including, falls (15.2%, n=13174), struck by/against (6.9%, n=5960), cuts (5.6%, n=4833), road traffic collisions (2.5%, n=2124) and poisoning (1%, n=900). We are continuing to refine and validate the categories using precision and recall and with comparison with coded diagnoses. Conclusions and Implications NLP can be used to extract valuable additional detailed information from ED data. We will further develop the algorithms for all hospitals in Wales for data from 2023 and design dashboards to produce a near-real time surveillance system.

An innovative linked Electronic Health Record derived data ecosystem for understanding the complexities of health and ageing.

ObjectiveWe describe the creation of an Electronic Health Record (EHR) derived linked data platform within the Australian National Centre for Healthy Ageing (NCHA). ApproachOur approach involved establishing a core set of EHR data suitable for research from the sole public health provider within a defined geographic region. Relevant items were selected based on published literature and consensus processes and curated within a specialised research data warehouse. Curation involved: data validation, quality assessment, internal linkage between episodes and data harmonisation/merging. An AI supported natural language processing pipeline was implemented to extract text-based data. Approvals were obtained from government agencies to link commonwealth claims data (primary care, medication, aged care, death) and state administrative hospital data, for all residents aged ≥60 years. Publicly available datasets were scoped for inclusion and collaborations established to incorporate local council data. ResultsA core of 113 curated items for >900,000 people over 10 years have been established across 4 hospitals and 10 community/outpatient services. Linked state and commonwealth data were obtained for 179,089 residents aged >60 years (Jan 2010-May 2021, linkage accuracy 98.4%). Environmental (greenery, air pollution, walkability) and Census data have been incorporated for each neighbourhood (Statistical Area 1). Data access, extraction and release has been tested through over 30 use-case projects in: dementia, residential aged care, medication use, homelessness, environmental impacts on health and ageing and health service redesign. ImplicationsThe NCHA Data Platform provides an international exemplar for the use of linked EHR data in advancing population health research.

A Comprehensive Natural Language Processing Pipeline for the Chronic Lupus Disease.

Electronic Health Records (EHRs) contain a wealth of unstructured patient data, making it challenging for physicians to do informed decisions. In this paper, we introduce a Natural Language Processing (NLP) approach for the extraction of therapies, diagnosis, and symptoms from ambulatory EHRs of patients with chronic Lupus disease. We aim to demonstrate the effort of a comprehensive pipeline where a rule-based system is combined with text segmentation, transformer-based topic analysis and clinical ontology, in order to enhance text preprocessing and automate rules' identification. Our approach is applied on a sub-cohort of 56 patients, with a total of 750 EHRs written in Italian language, achieving an Accuracy and an F-score over 97% and 90% respectively, in the three extracted domains. This work has the potential to be integrated with EHR systems to automate information extraction, minimizing the human intervention, and providing personalized digital solutions in the chronic Lupus disease domain.

Utilizing natural language processing to analyze student narrative reflections for medical curriculum improvement.

Medical curricula improvement is an ongoing process to keep material relevant and improve the student's learning experience to better prepare them for patient care. Many programs utilize end-of-year evaluations, but these frequently have low response rates and lack actionable feedback. We hypothesized that student reflections written during a fourth year Sub-Internship could be used retrospectively to mine additional information as feedback for future curriculum adjustments. However, reflections contain a large amount of narrative content that would require a cumbersome and essentially infeasible manual review process for busy medical education faculty. We developed a Natural Language Processing (NLP) pipeline to automatically identify common themes and topics present in the set of reflective writings that could be used to improve the curriculum. The dataset contains required responses to a faculty issued question submitted between August 2016 and July 2018 about challenges experienced during the medical students fourth year Sub-Internship. Eleven distinct topics were identified, with several being subsequently addressed in future iterations of the curriculum. Utilizing NLP on reflective writings was able to identify areas of curriculum improvement, and the NLP results provided a quick and easy way to explore the main themes and challenges expressed by students.

Development of a natural language processing pipeline for assessment of cardiovascular risk in myeloproliferative neoplasms.

Development of a natural language processing pipeline for assessment of cardiovascular risk in myeloproliferative neoplasms.

Use of Natural Language Processing to Extract and Classify Papillary Thyroid Cancer Features From Surgical Pathology Reports

BackgroundWe aim to use Natural Language Processing to automate the extraction and classification of thyroid cancer risk factors from pathology reports. MethodsWe analyzed 1410 surgical pathology reports from adult papillary thyroid cancer patients from 2010 to 2019. Structured and nonstructured reports were used to create a consensus-based ground truth dictionary and categorized them into modified recurrence risk levels. Nonstructured reports were narrative, while structured reports followed standardized formats. We developed ThyroPath, a rule-based Natural Language Processing pipeline, to extract and classify thyroid cancer features into risk categories. Training involved 225 reports (150 structured, 75 unstructured), with testing on 170 reports (120 structured, 50 unstructured) for evaluation. The pipeline's performance was assessed using both strict and lenient criteria for accuracy, precision, recall, and F1-score; a metric that combines precision and recall evaluation. ResultsIn extraction tasks, ThyroPath achieved overall strict F-1 scores of 93% for structured reports and 90% for unstructured reports, covering 18 thyroid cancer pathology features. In classification tasks, ThyroPath-extracted information demonstrated an overall accuracy of 93% in categorizing reports based on their corresponding guideline-based risk of recurrence: 76.9% for high-risk, 86.8% for intermediate risk, and 100% for both low and very low-risk cases. However, ThyroPath achieved 100% accuracy across all risk categories with human extracted pathology information. ConclusionsThyroPath shows promise in automating the extraction and risk recurrence classification of thyroid pathology reports at large scale. It offers a solution to laborious manual reviews and advancing virtual registries. However, it requires further validation before implementation.

Improving biomedical entity linking for complex entity mentions with LLM-based text simplification.

Large amounts of important medical information are captured in free-text documents in biomedical research and within healthcare systems, which can be made accessible through natural language processing (NLP). A key component in most biomedical NLP pipelines is entity linking, i.e. grounding textual mentions of named entities to a reference of medical concepts, usually derived from a terminology system, such as the Systematized Nomenclature of Medicine Clinical Terms. However, complex entity mentions, spanning multiple tokens, are notoriously hard to normalize due to the difficulty of finding appropriate candidate concepts. In this work, we propose an approach to preprocess such mentions for candidate generation, building upon recent advances in text simplification with generative large language models. We evaluate the feasibility of our method in the context of the entity linking track of the BioCreative VIII SympTEMIST shared task. We find that instructing the latest Generative Pre-trained Transformer model with a few-shot prompt for text simplification results in mention spans that are easier to normalize. Thus, we can improve recall during candidate generation by 2.9percentage points compared to our baseline system, which achieved the best score in the original shared task evaluation. Furthermore, we show that this improvement in recall can be fully translated into top-1 accuracy through careful initialization of a subsequent reranking model. Our best system achieves an accuracy of 63.6% on the SympTEMIST test set. The proposed approach has been integrated into the open-source xMEN toolkit, which is available online via https://github.com/hpi-dhc/xmen.

Demystifying large language models in second language development research

Evaluating students' textual response is a common and critical task in language research and education practice. However, manual assessment can be tedious and may lack consistency, posing challenges for both scientific discovery and frontline teaching. Leveraging state-of-the-art large language models (LLMs), we aim to define and operationalize LLM-Surprisal, a numeric representation of the interplay between lexical diversity and syntactic complexity, and to empirically and theoretically demonstrate its relevance for automatic writing assessment and Chinese L2 (second language) learners’ English writing development. We developed an LLM-based natural language processing pipeline that can automatically compute text Surprisal scores. By comparing Surprisal metrics with the widely used classic indices in L2 studies, we extended the usage of computational metrics in Chinese learners’ L2 English writing. Our analyses suggested that LLM-Surprisals can distinguish L2 from L1 (first language) writing, index L2 development stages, and predict scores provided by human professionals. This indicated that the Surprisal dimension may manifest itself as critical aspects in L2 development. The relative advantages and disadvantages of these approaches were discussed in depth. We concluded that LLMs are promising tools that can enhance L2 research. Our showcase paves the way for more nuanced approaches to computationally assessing and understanding L2 development. Our pipelines and findings will inspire language teachers, learners, and researchers to operationalize LLMs in an innovative and accessible manner.

Representation of Social Determinants of Health terminology in medical subject headings: impact of added terms.

To enhance and evaluate the quality of PubMed search results for Social Determinants of Health (SDoH) through the addition of new SDoH terms to Medical Subject Headings (MeSH). High priority SDoH terms and definitions were collated from authoritative sources, curated based on publication frequencies, and refined by subject matter experts. Descriptive analyses were used to investigate how PubMed search details and best match results were affected by the addition of SDoH concepts to MeSH. Three information retrieval metrics (Precision, Recall, and F measure) were used to quantitatively assess the accuracy of PubMed search results. Pre- and post-update documents were clustered into topic areas using a Natural Language Processing pipeline, and SDoH relevancy assessed. Addition of 35 SDoH terms to MeSH resulted in more accurate algorithmic translations of search terms and more reliable best match results. The Precision, Recall, and F measures of post-update results were significantly higher than those of pre-update results. The percentage of retrieved publications belonging to SDoH clusters was significantly greater in the post- than pre-update searches. This evaluation confirms that inclusion of new SDoH terms in MeSH can lead to qualitative and quantitative enhancements in PubMed search retrievals. It demonstrates the methodology for and impact of suggesting new terms for MeSH indexing. It provides a foundation for future efforts across behavioral and social science research (BSSR) domains. Improving the representation of BSSR terminology in MeSH can improve PubMed search results, thereby enhancing the ability of investigators and clinicians to build and utilize a cumulative BSSR knowledge base.

Combining Federated Machine Learning and Qualitative Methods to Investigate Novel Pediatric Asthma Subtypes: Protocol for a Mixed Methods Study.

Pediatric asthma is a heterogeneous disease; however, current characterizations of its subtypes are limited. Machine learning (ML) methods are well-suited for identifying subtypes. In particular, deep neural networks can learn patient representations by leveraging longitudinal information captured in electronic health records (EHRs) while considering future outcomes. However, the traditional approach for subtype analysis requires large amounts of EHR data, which may contain protected health information causing potential concerns regarding patient privacy. Federated learning is the key technology to address privacy concerns while preserving the accuracy and performance of ML algorithms. Federated learning could enable multisite development and implementation of ML algorithms to facilitate the translation of artificial intelligence into clinical practice. The aim of this study is to develop a research protocol for implementation of federated ML across a large clinical research network to identify and discover pediatric asthma subtypes and their progression over time. This mixed methods study uses data and clinicians from the OneFlorida+ clinical research network, which is a large regional network covering linked and longitudinal patient-level real-world data (RWD) of over 20 million patients from Florida, Georgia, and Alabama in the United States. To characterize the subtypes, we will use OneFlorida+ data from 2011 to 2023 and develop a research-grade pediatric asthma computable phenotype and clinical natural language processing pipeline to identify pediatric patients with asthma aged 2-18 years. We will then apply federated learning to characterize pediatric asthma subtypes and their temporal progression. Using the Promoting Action on Research Implementation in Health Services framework, we will conduct focus groups with practicing pediatric asthma clinicians within the OneFlorida+ network to investigate the clinical utility of the subtypes. With a user-centered design, we will create prototypes to visualize the subtypes in the EHR to best assist with the clinical management of children with asthma. OneFlorida+ data from 2011 to 2023 have been collected for 411,628 patients aged 2-18 years along with 11,156,148 clinical notes. We expect to complete the computable phenotyping within the first year of the project, followed by subtyping during the second and third years, and then will perform the focus groups and establish the user-centered design in the fourth and fifth years of the project. Pediatric asthma subtypes incorporating RWD from diverse populations could improve patient outcomes by moving the field closer to precision pediatric asthma care. Our privacy-preserving federated learning methodology and qualitative implementation work will address several challenges of applying ML to large, multicenter RWD data. DERR1-10.2196/57981.

Clinical efficacy of pre-trained large language models through the lens of aphasia

The rapid development of large language models (LLMs) motivates us to explore how such state-of-the-art natural language processing systems can inform aphasia research. What kind of language indices can we derive from a pre-trained LLM? How do they differ from or relate to the existing language features in aphasia? To what extent can LLMs serve as an interpretable and effective diagnostic and measurement tool in a clinical context? To investigate these questions, we constructed predictive and correlational models, which utilize mean surprisals from LLMs as predictor variables. Using AphasiaBank archived data, we validated our models’ efficacy in aphasia diagnosis, measurement, and prediction. Our finding is that LLMs-surprisals can effectively detect the presence of aphasia and different natures of the disorder, LLMs in conjunction with the existing language indices improve models’ efficacy in subtyping aphasia, and LLMs-surprisals can capture common agrammatic deficits at both word and sentence level. Overall, LLMs have potential to advance automatic and precise aphasia prediction. A natural language processing pipeline can be greatly benefitted from integrating LLMs, enabling us to refine models of existing language disorders, such as aphasia.

Early evaluation of a natural language processing tool to improve access to educational resources for surgical patients.

Accessible patient information sources are vital in educating patients about the benefits and risks of spinal surgery, which is crucial for obtaining informed consent. We aim to assess the effectiveness of a natural language processing (NLP) pipeline in recognizing surgical procedures from clinic letters and linking this with educational resources. Retrospective examination of letters from patients seeking surgery for degenerative spinal disease at a single neurosurgical center. We utilized MedCAT, a named entity recognition and linking NLP, integrated into the electronic health record (EHR), which extracts concepts and links them to systematized nomenclature of medicine-clinical terms (SNOMED-CT). Investigators reviewed clinic letters, identifying words or phrases that described or identified operations and recording the SNOMED-CT terms as ground truth. This was compared to SNOMED-CT terms identified by the model, untrained on our dataset. A pipeline linking clinic letters to patient-specific educational resources was established, and precision, recall, and F1 scores were calculated. Across 199 letters the model identified 582 surgical procedures, and the overall pipeline after adding rules a total of 784 procedures (precision = 0.94, recall = 0.86, F1 = 0.91). Across 187 letters with identified SNOMED-CT terms the integrated pipeline linking education resources directly to the EHR was successful in 157 (78%) patients (precision = 0.99, recall = 0.87, F1 = 0.92). NLP accurately identifies surgical procedures in pre-operative clinic letters within an untrained subspecialty. Performance varies among letter authors and depends on the language used by clinicians. The identified procedures can be linked to patient education resources, potentially improving patients' understanding of surgical procedures.

Advancing equity in breast cancer care: natural language processing for analysing treatment outcomes in under-represented populations

ObjectiveThe study aimed to develop natural language processing (NLP) algorithms to automate extracting patient-centred breast cancer treatment outcomes from clinical notes in electronic health records (EHRs), particularly for women from...

Age modulates the predictive value of self-reported sleepiness for all-cause mortality risk: insights from a comprehensive national database of veterans.

Excessive daytime sleepiness (EDS) is prevalent and overwhelmingly stems from disturbed sleep. We hypothesized that age modulates the association between EDS and increased all-cause mortality. We utilized the Veterans' Health Administration data from 1999-2022. We enrolled participants with sleep related ICD9/10 codes or sleep services. A natural language processing (NLP) pipeline was developed and validated to extract the Epworth Sleepiness Scale (ESS) as a self-reported tool to measure EDS from physician progress notes. The NLP's accuracy was assessed through manual annotation of 470 notes. Participants were categorized into Normal-ESS, n-ESS, (ESS 0-10) and high-ESS, h-ESS, (ESS 11-24). We created three age groups: < 50 years; 50 to < 65 years; and ≥ 65 years. The adjusted odds ratio (aOR) of mortality was calculated for age, BMI, sex, race, ethnicity, and the Charlson Comorbidity Index (CCI), using n-ESS as the reference. Subsequently, we conducted age stratified analysis. The first ESS records were extracted from 423,087 veterans with a mean age of 54.8 (±14.6), mean BMI of 32.6 (±6.2), and 90.5% male. The aOR across all ages was 17% higher (1.15,1.19) in the h-ESS category. The aORs only became statistically significant for individuals aged ≥ 50 years in the h-ESS compared to the n-ESS category (< 50 years: 1.02 [0.96,1.08], 50 to < 65 years 1.13[1.10,1.16]; ≥ 65 years: 1.25 [1.21-1.28]). High ESS, predicted increased mortality only in participants aged 50 and older. Further research is required to identify this differential behavior in relation to age.

Age modifies the association between severe sleep apnea and all-cause mortality

PurposeWhile sleep apnea (SA) gets more prevalent with advancing age, the impact of age on the association between SA and health outcomes is not well known. We assessed the association between the severity of SA and all-cause mortality in different age groups using large longitudinal data. MethodWe applied a Natural Language Processing pipeline to extract the apnea-hypopnea index (AHI) from the physicians’ interpretation of sleep studies performed at the Veteran Health Administration (FY 1999–2022). We categorized the participants as no SA (n-SA, AHI< 5) and severe SA (s-SA, AHI≥30). We grouped the cohort based on age: Young≤40; Middle-aged:40–65; and Older adults≥65; and calculated the odds ratio (aOR) of mortality adjusted for age, sex, race, ethnicity, BMI, and Charlson-Comorbidity Index (CCI) using n-SA as the reference. ResultsWe identified 146,148 participants (age 52.23 ± 15.02; BMI 32.11 ± 6.05; male 86.7 %; White 66 %). Prevalence of s-SA increased with age. All-cause mortality was lower in s-SA compared to n-SA in the entire cohort (aOR,0.56; 95%CI: 0.54,0.58). Comparing s-SA to n-SA, the all-cause mortality rates (Young 1.86 % vs 1.49 %; Middle-aged 12.07 % vs 13.34 %; and Older adults 26.35 % vs 40.18 %) and the aOR diminished as the age increased (Young: 1.11, 95%CI: 0.93–1.32; Middle-aged: 0.64, 95%CI: 0.61–0.67; and Older adults: 0.44, 95%CI: 0.41–0.46). ConclusionThe prevalence of severe SA increased while the odds of all-cause mortality compared to n-SA diminished with age. SA may exert less harmful effects on the aged population. A causality analysis is warranted to assess the relationship between SA, aging, and all-cause mortality.

Natural language processing pipeline to extract prostate cancer-related information from clinical notes.

To develop an automated pipeline for extracting prostate cancer-related information from clinical notes. This retrospective study included 23,225 patients who underwent prostate MRI between 2017 and 2022. Cancer risk factors (family history of cancer and digital rectal exam findings), pre-MRI prostate pathology, and treatment history of prostate cancer were extracted from free-text clinical notes in English as binary or multi-class classification tasks. Any sentence containing pre-defined keywords was extracted from clinical notes within one year before the MRI. After manually creating sentence-level datasets with ground truth, Bidirectional Encoder Representations from Transformers (BERT)-based sentence-level models were fine-tuned using the extracted sentence as input and the category as output. The patient-level output was determined by compilation of multiple sentence-level outputs using tree-based models. Sentence-level classification performance was evaluated using the area under the receiver operating characteristic curve (AUC) on 15% of the sentence-level dataset (sentence-level test set). The patient-level classification performance was evaluated on the patient-level test set created by radiologists by reviewing the clinical notes of 603 patients. Accuracy and sensitivity were compared between the pipeline and radiologists. Sentence-level AUCs were ≥ 0.94. The pipeline showed higher patient-level sensitivity for extracting cancer risk factors (e.g., family history of prostate cancer, 96.5% vs. 77.9%, p < 0.001), but lower accuracy in classifying pre-MRI prostate pathology (92.5% vs. 95.9%, p = 0.002) and treatment history of prostate cancer (95.5% vs. 97.7%, p = 0.03) than radiologists, respectively. The proposed pipeline showed promising performance, especially for extracting cancer risk factors from patient's clinical notes. The natural language processing pipeline showed a higher sensitivity for extracting prostate cancer risk factors than radiologists and may help efficiently gather relevant text information when interpreting prostate MRI. When interpreting prostate MRI, it is necessary to extract prostate cancer-related information from clinical notes. This pipeline extracted the presence of prostate cancer risk factors with higher sensitivity than radiologists. Natural language processing may help radiologists efficiently gather relevant prostate cancer-related text information.

Novel approach to implementing natural language processing for clinical staging of non-small-cell lung cancer.

e13624 Background: Cancer staging is instrumental in driving clinical management and trial enrollment, but staging data are generally unreliable and unstructured in the electronic health record (EHR). Advances in natural language processing (NLP) may facilitate clinical staging and documentation [1], but challenges to real-world implementation include (1) automatically identifying appropriate patients and reports from the EHR and (2) developing an unbiased dataset for training and validation [2]. We describe our institution’s novel approach to overcome these barriers while building an in-house NLP pipeline for clinical tumor staging of non-small-cell lung cancer (NSCLC). Methods: We identified patients by searching our EHR (Epic) for a molecular analysis test ordered specifically for pathological diagnoses of NSCLC at our institution. We used the test order date as the diagnosis proxy date (DPD). For each patient, we extracted imaging reports up to 16 weeks before and 6 weeks after the DPD. To derive primary tumor size, we analyzed the CT Chest or PET/CT report closest to the DPD using an oncology-trained NLP text extraction and labeling tool (John Snow Labs). We cleaned all extracted tumor size entities and identified the largest measurement linked to the lungs. We compared primary tumor measurements from the NLP pipeline to those in a preexisting, manually compiled cancer registry (CNEXT). We manually analyzed discrepancies through chart review. Results: 542 patients with a DPD between 11/2016 - 9/2023 were processed through the NLP pipeline. Of 443 patients with valid values in both the pipeline and CNEXT, 53% (234) were exact matches, and 20% (90) had a close match (within 0-5mm), yielding a 73% accuracy rate for values within 5mm. When mismatched values were manually reviewed, several cases in CNEXT were found to have a DPD differing by more than 3 months and tumor sizes derived from external reports. When these cases were excluded, 320 of the remaining 349 patients had valid values in both the pipeline and the updated manual review. In this refined population, 66% (213) were exact matches, and 15% (48) had a close match, yielding an 82% accuracy rate for values within 5mm. Conclusions: To our knowledge, this is the first report of a pathology-based method to automatically and reliably identify patients with NSCLC and their relevant imaging reports directly from the EHR. We used a prebuilt NLP tool to derive primary tumor sizes with relatively high accuracy and found that adding flags for timeline discrepancies and external reports can further improve validity. As we near completion of analogous pipelines for node and metastasis staging, we will develop methodology to identify subgroups of patients that can be clinically staged with near-perfect accuracy, ultimately aiming to substantially limit manual staging of uncomplicated cases. 1. Puts 2023. 2. Wang 2022.

Implementing a natural language processing pipeline to expedite clinical chart review to assess germline genetic testing in veterans with breast cancer.

e13640 Background: The Making Advances in Mammography and Medical Options for Veterans Act of 2022 mandated that the Department of Veterans Affairs (VA) perform a study of germline genetic testing in Veterans with breast cancer (BC). Assessing whether a Veteran with BC was offered germline genetic testing based on electronic health records requires time-consuming review of unstructured clinical notes by clinical experts who are also extremely busy providing care. To expedite this review process, we implemented a productionized natural language processing (NLP) pipeline to identify relevant concepts and highlight them within a custom-made chart review interface. Methods: Clinical subject-matter experts (SMEs) were convened to identify keywords suggestive of genetic testing. These were categorized into 3 sets: gene-related concepts, VA-specific services, and exams and tests. We then used entity recognition and entity linking to link these concepts to a knowledge base. These were enriched with entities similarly extracted from an open-source reference clinical text on germline genetic testing in BC. The resulting concepts were visualized as a word cloud and reviewed by the SMEs. We implemented a Power BI and Power Apps interface to display clinical notes from 200 randomly selected Veterans, with relevant concepts highlighted. Each Veteran’s notes were reviewed independently by 2 of the 5 SMEs, who assessed whether the Veteran was offered testing, referred to a genetic counseling service, or otherwise had evidence of testing. Disagreements were adjudicated by the whole team. Results: 11622 notes (mean 58/Veteran, 3028 characters/note) were reviewed. Table shows the most prevalent entities among genes, mutations, and exam names. Of the 5 reviewers, 3 annotated 80 patients each in a single session, lasting less than 50 minutes on average. Reviewers had an inter-annotator agreement of 85.5%. Mention of target concepts was associated with germline genetic testing with an accuracy of 80.4%, precision of 76.4%, recall of 99.1%, and F1 of 86.3%. Germline genetic testing rates are detailed in a separate submission. Conclusions: Modern informatics tools such as productionized NLP, the Microsoft Power Platform, and high-performance computing environments allow for rapid implementation of health information technology tools that can enable clinicians to gain critical insights quickly and efficiently from electronic health record data. [Table: see text]