Rule-based Natural Language Processing Research Articles

Classifying Individuals With Rheumatic Conditions as Financially Insecure Using Electronic Health Record Data and Natural Language Processing: Algorithm Derivation and Validation.

We aimed to examine the feasibility of applying natural language processing (NLP) to unstructured electronic health record (EHR) documents to detect the presence of financial insecurity among patients with rheumatologic disease enrolled in an integrated care management program (iCMP). We incorporated supervised, rule-based NLP and statistical methods to identify financial insecurity among patients with rheumatic conditions enrolled in an iCMP (n = 20,395) in a multihospital EHR system. We constructed a lexicon for financial insecurity using data from available knowledge sources and then reviewed EHR notes from 538 randomly selected individuals (training cohort n = 366, validation cohort n = 172). We manually categorized records as having "definite," "possible," or "no" mention of financial insecurity. All available notes were processed using Narrative Information Linear Extraction, a rule-based version of NLP. Models were trained using the NLP features for financial insecurity using logistic, least absolute shrinkage operator (LASSO), and random forest performance characteristic and were compared with the reference standard. A total of 245,142 notes were processed from 538 individual patient records. Financial insecurity was present among 100 (27%) individuals in the training cohort and 63 (37%) in the validation cohort. The LASSO and random forest models performed identically and slightly better than logistic regression, with positive predictive values of 0.90, sensitivities of 0.29, and specificities of 0.98. The development of a context-driven lexicon used with rule-based NLP to extract data that identify financial insecurity is feasible for use and improved the capture for presence of financial insecurity with high accuracy. In the absence of a standard lexicon and construct definition for financial insecurity status, additional studies are needed to optimize the sensitivity of algorithms to categorize financial insecurity with construct validity.

Understanding older people's voice interactions with smart voice assistants: a new modified rule-based natural language processing model with human input

The COVID-19 pandemic has expedited the integration of Smart Voice Assistants (SVA) among older people. The qualitative data derived from user commands on SVA is pivotal for elucidating the engagement patterns of older individuals with such systems. However, the sheer volume of user-generated voice interaction data presents a formidable challenge for manual coding. Compounding this issue, age-related cognitive decline and alterations in speech patterns further complicate the interpretation of older users’ SVA voice interactions. Conventional dictionary-based textual analysis tools, which count word frequencies, are inadequate in capturing the evolving and communicative essence of these interactions that unfold over a series of dialogues and modify with time. To address these challenges, our study introduces a novel, modified rule-based Natural Language Processing (MR-NLP) model augmented with human input. This reproducible approach capitalizes on human-derived insights to establish a lexicon of critical keywords and to formulate rules for the iterative refinement of the NLP model. English speakers, aged 50 or older and residing alone, were enlisted to engage with Amazon Alexa™ via predefined daily routines for a minimum of 30 min daily spanning three months (N = 35, mean age = 77). We amassed time-stamped, textual data comprising participants’ user commands and responses from Alexa™. Initially, a subset constituting 20% of the data (1,020 instances) underwent manual coding by human coder, predicated on keywords and commands. Separately, a rule-based Natural Language Processing (NLP) methodology was employed to code the identical subset. Discrepancies arising between human coder and the NLP model programmer were deliberated upon and reconciled to refine the rule-based NLP coding framework for the entire dataset. The modified rule-based NLP approach demonstrated notable enhancements in efficiency and scalability and reduced susceptibility to inadvertent errors in comparison to manual coding. Furthermore, human input was instrumental in augmenting the NLP model, yielding insights germane to the aging adult demographic, such as recurring speech patterns or ambiguities. By disseminating this innovative software solution to the scientific community, we endeavor to advance research and innovation in NLP model formulation, subsequently contributing to the understanding of older people's interactions with SVA and other AI-powered systems.

Identification of pancreatic cancer risk factors from clinical notes using natural language processing

ObjectivesScreening for pancreatic ductal adenocarcinoma (PDAC) is considered in high-risk individuals (HRIs) with established PDAC risk factors, such as family history and germline mutations in PDAC susceptibility genes. Accurate assessment of risk factor status is provider knowledge-dependent and requires extensive manual chart review by experts. Natural Language Processing (NLP) has shown promise in automated data extraction from the electronic health record (EHR). We aimed to use NLP for automated extraction of PDAC risk factors from unstructured clinical notes in the EHR. MethodsWe first developed rule-based NLP algorithms to extract PDAC risk factors at the document-level, using an annotated corpus of 2091 clinical notes. Next, we further improved the NLP algorithms using a cohort of 1138 patients through patient-level training, validation, and testing, with comparison against a pre-specified reference standard. To minimize false-negative results we prioritized algorithm recall. ResultsIn the test set (n = 807), the NLP algorithms achieved a recall of 0.933, precision of 0.790, and F1-score of 0.856 for family history of PDAC. For germline genetic mutations, the algorithm had a high recall of 0.851, while precision and F1-score were lower at 0.350 and 0.496 respectively. Most false positives for germline mutations resulted from erroneous recognition of tissue mutations. ConclusionsRule-based NLP algorithms applied to unstructured clinical notes are highly sensitive for automated identification of PDAC risk factors. Further validation in a large primary-care patient population is warranted to assess real-world utility in identifying HRIs for pancreatic cancer screening.

Rule-Based Natural Language Processing Pipeline to Detect Medication-Related Named Entities: Insights for Transfer Learning.

We document the procedure and performance of a rule-based NLP system that, using transfer learning, automatically extracts essential named entities related to drug errors from Japanese free-text incident reports. Subsequently, we used the rule-based annotated data to fine-tune a pre-trained BERT model and examined the performance of medication-related incident report prediction. The rule-based pipeline achieved a macro-F1-score of 0.81 in an internal dataset and the BERT model fine-tuned with rule-annotated data achieved a macro-F1-score of 0.97 and 0.75 for named entity recognition and relation extraction tasks, respectively. The model can be deployed to other, similar problems in medication-related clinical texts.

Proposal of a new model of national skin audit and data submission.

This article discusses the role of multidisciplinary teams in improving data flow to cancer registries and the inefficiencies of current retrospective data entry methods. It proposes a novel model using a rule-based natural language processing technique for basal cell carcinoma, integrating real-time analysis and structured data presentation to enhance national skin audit and data submission. The model faces challenges like varied reporting styles and the need for common ontologies but aims to fill a gap in dermatology and plastic surgery audits, improving healthcare quality through precise and timely data analysis.

Natural Language Processing Algorithm Used for Staging Pulmonary Oncology from Free-Text Radiological Reports: “Including PET-CT and Validation Towards Clinical Use”

Natural language processing (NLP) can be used to process and structure free text, such as (free text) radiological reports. In radiology, it is important that reports are complete and accurate for clinical staging of, for instance, pulmonary oncology. A computed tomography (CT) or positron emission tomography (PET)-CT scan is of great importance in tumor staging, and NLP may be of additional value to the radiological report when used in the staging process as it may be able to extract the T and N stage of the 8th tumor–node–metastasis (TNM) classification system. The purpose of this study is to evaluate a new TN algorithm (TN-PET-CT) by adding a layer of metabolic activity to an already existing rule-based NLP algorithm (TN-CT). This new TN-PET-CT algorithm is capable of staging chest CT examinations as well as PET-CT scans. The study design made it possible to perform a subgroup analysis to test the external validation of the prior TN-CT algorithm. For information extraction and matching, pyContextNLP, SpaCy, and regular expressions were used. Overall TN accuracy score of the TN-PET-CT algorithm was 0.73 and 0.62 in the training and validation set (N = 63, N = 100). The external validation of the TN-CT classifier (N = 65) was 0.72. Overall, it is possible to adjust the TN-CT algorithm into a TN-PET-CT algorithm. However, outcomes highly depend on the accuracy of the report, the used vocabulary, and its context to express, for example, uncertainty. This is true for both the adjusted PET-CT algorithm and for the CT algorithm when applied in another hospital.

Bridging information gaps in menopause status classification through natural language processing.

To use natural language processing (NLP) of clinical notes to augment existing structured electronic health record (EHR) data for classification of a patient's menopausal status. A rule-based NLP system was designed to capture evidence of a patient's menopause status including dates of a patient's last menstrual period, reproductive surgeries, and postmenopause diagnosis as well as their use of birth control and menstrual interruptions. NLP-derived output was used in combination with structured EHR data to classify a patient's menopausal status. NLP processing and patient classification were performed on a cohort of 307512 female Veterans receiving healthcare at the US Department of Veterans Affairs (VA). NLP was validated at 99.6% precision. Including the NLP-derived data into a menopause phenotype increased the number of patients with data relevant to their menopausal status by 118%. Using structured codes alone, 81173 (27.0%) are able to be classified as postmenopausal or premenopausal. However, with the inclusion of NLP, this number increased 167804 (54.6%) patients. The premenopausal category grew by 532.7% with the inclusion of NLP data. By employing NLP, it became possible to identify documented data elements that predate VA care, originate outside VA networks, or have no corresponding structured field in the VA EHR that would be otherwise inaccessible for further analysis. NLP can be used to identify concepts relevant to a patient's menopausal status in clinical notes. Adding NLP-derived data to an algorithm classifying a patient's menopausal status significantly increases the number of patients classified using EHR data, ultimately enabling more detailed assessments of the impact of menopause on health outcomes.

Automated-detection of risky alcohol use prior to surgery using natural language processing.

Preoperative risky alcohol use is one of the most common surgical risk factors. Accurate and early identification of risky alcohol use could enhance surgical safety. Artificial Intelligence-based approaches, such as natural language processing (NLP), provide an innovative method to identify alcohol-related risks from patients' electronic health records (EHR) before surgery. Clinical notes (n = 53,629) from pre-operative patients in a tertiary care facility were analyzed for evidence of risky alcohol use and alcohol use disorder. One hundred of these records were reviewed by experts and labeled for comparison. A rule-based NLP model was built, and we assessed the clinical notes for the entire population. Additionally, we assessed each record for the presence or absence of alcohol-related International Classification of Diseases (ICD) diagnosis codes as an additional comparator. NLP correctly identified 87% of the human-labeled patients classified with risky alcohol use. In contrast, diagnosis codes alone correctly identified only 29% of these patients. In terms of specificity, NLP correctly identified 84% of the non-risky cohort, while diagnosis codes correctly identified 90% of this cohort. In the analysis of the full dataset, the NLP-based approach identified three times more patients with risky alcohol use than ICD codes. NLP, an artificial intelligence-based approach, efficiently and accurately identifies alcohol-related risk in patients' EHRs. This approach could supplement other alcohol screening tools to identify patients in need of intervention, treatment, and/or postoperative withdrawal prophylaxis. Alcohol-related ICD diagnosis had limited utility relative to NLP, which extracts richer information within clinical notes to classify patients.

A Transformer Natural Language Processing Algorithm for Cancer Associated Thrombosis Phenotype

Introduction: Cancer patients have high risk of venous thromboembolism (VTE). The ability to accurately capture VTE outcome in large electronic health record (EHR) is critical for both surveillance and risk stratification. We previously published a rule-based natural language processing (NLP) algorithm that relied on radiology reports alone with a precision (positive predictive value) of 89% and recall (sensitivity) of 69-78% (sTable 5 PMID36626707). The low recall metric was expected as VTE diagnosis that occur at outside hospital would not be captured by radiology reports. In the current study, we aimed to assess the performance of a machine learning NLP algorithm using both radiology and medical notes to detect cancer associated VTE longitudinally in a large cancer cohort. Methods: VTEwasdefined as symptomatic or incidental pulmonary embolism, lower or upper extremity deep vein thrombosis. Unusual thrombosis such as splanchnic vein thrombosis (mostly tumor thrombi) was classified as negative. Figure 1 details the study design: i) patient selection, ii) pre-annotation filtering, iii) gold standard annotation, iv) transformer model development. From a cohort of 9,769 patients with active cancer receiving first-line systemic therapy, we first pre-filtered all clinic progress notes, discharge summaries, and radiology reports to exclude sections with little clinical relevance (e.g. review of systems, physical exam), while maintaining sections with high clinical value (e.g. history of present illness, assessment/plan, hospital problems/course). Then, we kept notes with VTE keyword stems (e.g. VTE, PE, DVT, thromb, embol, clot, filling defect, phlebitis). We selected 808 patients with at least 1 keyword to determine the gold standard VTE date and location through independent chart review process. This cohort was randomly split into a training and testing cohort at an 7:1 ratio. In the training cohort (n=708), we further annotated all keyword-containing phrases within 30 days (d) from VTE date and from index date until 180d if no VTE (n=~2,000). The ~800 positive and ~1,200 negative keyword-containing phrases were used to train ClinicalBERT (bidirectional encoder representation transformer) large language model through split-sample cross validation to finetune the loss function and to determine the prediction threshold. Finally, we applied the entire trained model package to every available notes/reports (n=~2300) in the test cohort (n=100) to assess if the first positively predicted date would correlate with the gold standard VTE date. Results: The finetuned ClinicalBERT modelachieved 98% precision and 98% recall on the patient level in the training cohort (n=708). When applied on the 100-patient test cohort, the model accurately predicted first VTE events in 50/54 patients (93% precision) and captured 50/54 of patients with VTE (93% recall sensitivity) (Table 1). Most of the tumor thrombi and superficial venous thrombi were correctly classified as negative. The false positives were related to unusual sentences with both positive and negative findings such as “Thrombosis surrounding the left brachial catheter in basilic vein,” “CT chest with PE was done,” “Lovenox was initiated for UE DVT, but this was port thrombosis in the basilic vein.” The false negatives were related to isolated VTE diagnosis close to death or loss to follow-up. Notably, the precision and recall tradeoff can be further finetuned by changing the minimum number of positively predicted notes from the NLP. Conclusion: In conclusion, we successfully developed and tested a transformer NLP model to detect cancer associated VTE longitudinally using a combination of clinical notes and radiology reports that achieved a precision of 93% and recall of 93%. This represented an improvement over our original algorithm with radiology report alone. Further validation of the model in external cohorts and finetuning through federated or transferred learning are ongoing to ensure model generalizability and usability.

Bridging Linguistic Divides: Rule-Based Marathi to Gujarati Text Translator (Part-I)

In our increasingly interconnected world, language diversity poses formidable challenges to effective communication and access to information. These challenges are particularly pronounced in culturally rich and linguistically diverse societies, such as India. This research paper presents an in-depth exploration of the development of a rule-based Marathi to Gujarati Text Translator, project sponsored by Mass Technologies. The primary objective of this project is to mitigate the language barriers that hinder effective communication and limit access to information. Through the application of rule-based Natural Language Processing (NLP) techniques, this translator aims to provide accurate and contextually relevant translations. This paper emphasizes the alignment of the project’s objectives with Mass Technologies’ organizational missions, which prioritize effective communication, cultural exchange, accessibility, and support for diverse societies. The paper delves into the intricacies of the project, including its goals, methodology, system requirements, future steps, risk management strategies, and its potential impact on society.

Application of natural language processing to post-structuring of rectal cancer MRI reports

To evaluate a natural language processing (NLP) system for extracting structured information from the free-form text of rectal cancer magnetic resonance imaging (MRI) reports written in Chinese. A rule-based NLP model that could extract 11 key image features of rectal cancer was constructed using 358 MRI reports of rectal cancer written between 2015 and 2021. Fifty reports written before 2015 and 50 written after 2021 were used as test datasets, and the reference standard was determined by manual extraction of information by two radiologists. The length and reporting rate of image features in pre-2015 and post-2021 datasets, as well as the accuracy, precision, recall, and F1 score of feature extraction by the NLP system, were compared. The time required for the NLP to extract data was compared with that required by the radiologists. Reports written after 2021 had longer diagnostic impression sections than reports written before 2015. The reporting rate of key imaging features of rectal cancer was 36.55% before 2015 and 79.82% after 2021. The accuracy, precision, recall, and F1 score of NLP for correct extraction of values from reports were 93.82%, 95.63%, 87.06%, and 91.15%, respectively, for pre-2015 reports, and 92.55%, 98.53%, 94.15%, and 96.29%, respectively, for post-2021 reports. NLP generated all the structured information in <1 second. The NLP system with rule-based pattern matching achieved rapid and accurate structured processing of rectal cancer MRI reports. MRI reports with structured templates are more suitable for NLP-based extraction of information.

Forms or Free-Text? Measuring Advance Care Planning Activity Using Electronic Health Records.

Advance care planning (ACP) discussions seek to guide future serious illness care. These discussions may be recorded in the electronic health record by documentation in clinical notes, structured forms and directives, and physician orders. Yet, most studies of ACP prevalence have only examined structured electronic health record elements and ignored data existing in notes. We sought to investigate the relative comprehensiveness and accuracy of ACP documentation from structured and unstructured electronic health record data sources. We evaluated structured and unstructured ACP documentation present in the electronic health records of 435 patients with cancer drawn from three separate healthcare systems. We extracted structured ACP documentation by manually annotating written documents and forms scanned into the electronic health record. We coded unstructured ACP documentation using a rule-based natural language processing software that identified ACP keywords within clinical notes and was subsequently reviewed for accuracy. The unstructured approach identified more instances of ACP documentation (238, 54.7% of patients) than the structured ACP approach (187, 42.9% of patients). Additionally, 16.6% of all patients with structured ACP documentation only had documents that were judged as misclassified, incomplete, blank, unavailable, or a duplicate of a previously entered erroneous document. ACP documents scanned into electronic health records represent a limited view of ACP activity. Research and measures of clinical practice with ACP should incorporate information from unstructured data.

EXSCLAIM!: Harnessing materials science literature for self-labeled microscopy datasets

This work introduces the EXSCLAIM! toolkit for the automatic extraction, separation, and caption-based natural language annotation of images from scientific literature. EXSCLAIM! is used to show how rule-based natural language processing and image recognition can be leveraged to construct an electron microscopy dataset containing thousands of keyword-annotated nanostructure images. Moreover, it is demonstrated how a combination of statistical topic modeling and semantic word similarity comparisons can be used to increase the number and variety of keyword annotations on top of the standard annotations from EXSCLAIM! With large-scale imaging datasets constructed from scientific literature, users are well positioned to train neural networks for classification and recognition tasks specific to microscopy-tasks often otherwise inhibited by a lack of sufficient annotated training data.

Understanding the performance and reliability of NLP tools: a comparison of four NLP tools predicting stroke phenotypes in radiology reports.

Natural language processing (NLP) has the potential to automate the reading of radiology reports, but there is a need to demonstrate that NLP methods are adaptable and reliable for use in real-world clinical applications. We tested the F1 score, precision, and recall to compare NLP tools on a cohort from a study on delirium using images and radiology reports from NHS Fife and a population-based cohort (Generation Scotland) that spans multiple National Health Service health boards. We compared four off-the-shelf rule-based and neural NLP tools (namely, EdIE-R, ALARM+, ESPRESSO, and Sem-EHR) and reported on their performance for three cerebrovascular phenotypes, namely, ischaemic stroke, small vessel disease (SVD), and atrophy. Clinical experts from the EdIE-R team defined phenotypes using labelling techniques developed in the development of EdIE-R, in conjunction with an expert researcher who read underlying images. EdIE-R obtained the highest F1 score in both cohorts for ischaemic stroke, ≥93%, followed by ALARM+, ≥87%. The F1 score of ESPRESSO was ≥74%, whilst that of Sem-EHR is ≥66%, although ESPRESSO had the highest precision in both cohorts, 90% and 98%. For F1 scores for SVD, EdIE-R scored ≥98% and ALARM+ ≥90%. ESPRESSO scored lowest with ≥77% and Sem-EHR ≥81%. In NHS Fife, F1 scores for atrophy by EdIE-R and ALARM+ were 99%, dropping in Generation Scotland to 96% for EdIE-R and 91% for ALARM+. Sem-EHR performed lowest for atrophy at 89% in NHS Fife and 73% in Generation Scotland. When comparing NLP tool output with brain image reads using F1 scores, ALARM+ scored 80%, outperforming EdIE-R at 66% in ischaemic stroke. For SVD, EdIE-R performed best, scoring 84%, with Sem-EHR 82%. For atrophy, EdIE-R and both ALARM+ versions were comparable at 80%. The four NLP tools show varying F1 (and precision/recall) scores across all three phenotypes, although more apparent for ischaemic stroke. If NLP tools are to be used in clinical settings, this cannot be performed "out of the box." It is essential to understand the context of their development to assess whether they are suitable for the task at hand or whether further training, re-training, or modification is required to adapt tools to the target task.

Artificial intelligence to identify fractures on pediatric and young adult upper extremity radiographs.

Pediatric fractures are challenging to identify given the different response of the pediatric skeleton to injury compared to adults, and most artificial intelligence (AI) fracture detection work has focused on adults. Develop and transparently share an AI model capable of detecting a range of pediatric upper extremity fractures. In total, 58,846 upper extremity radiographs (finger/hand, wrist/forearm, elbow, humerus, shoulder/clavicle) from 14,873 pediatric and young adult patients were divided into train (n = 12,232 patients), tune (n = 1,307), internal test (n = 819), and external test (n = 515) splits. Fracture was determined by manual inspection of all test radiographs and the subset of train/tune radiographs whose reports were classified fracture-positive by a rule-based natural language processing (NLP) algorithm. We trained an object detection model (Faster Region-based Convolutional Neural Network [R-CNN]; "strongly-supervised") and an image classification model (EfficientNetV2-Small; "weakly-supervised") to detect fractures using train/tune data and evaluate on test data. AI fracture detection accuracy was compared with accuracy of on-call residents on cases they preliminarily interpreted overnight. A strongly-supervised fracture detection AI model achieved overall test area under the receiver operating characteristic curve (AUC) of 0.96 (95% CI 0.95-0.97), accuracy 89.7% (95% CI 88.0-91.3%), sensitivity 90.8% (95% CI 88.5-93.1%), and specificity 88.7% (95% CI 86.4-91.0%), and outperformed a weakly-supervised model (AUC 0.93, 95% CI 0.92-0.94, P < 0.0001). AI accuracy on cases preliminary interpreted overnight was higher than resident accuracy (AI 89.4% vs. 85.1%, 95% CI 87.3-91.5% vs. 82.7-87.5%, P = 0.01). An object detection AI model identified pediatric upper extremity fractures with high accuracy.

Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer.

Background The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. Purpose To compare the performance of the LLMs ChatGPT and GPT-4 in data mining and labeling oncologic phenotypes from free-text CT reports on lung cancer by using user-defined prompts. Materials and Methods This retrospective study included patients who underwent lung cancer follow-up CT between September 2021 and March 2023. A subset of 25 reports was reserved for prompt engineering to instruct the LLMs in extracting lesion diameters, labeling metastatic disease, and assessing oncologic progression. This output was fed into a rule-based natural language processing pipeline to match ground truth annotations from four radiologists and derive performance metrics. The oncologic reasoning of LLMs was rated on a five-point Likert scale for factual correctness and accuracy. The occurrence of confabulations was recorded. Statistical analyses included Wilcoxon signed rank and McNemar tests. Results On 424 CT reports from 424 patients (mean age, 65 years ± 11 [SD]; 265 male), GPT-4 outperformed ChatGPT in extracting lesion parameters (98.6% vs 84.0%, P < .001), resulting in 96% correctly mined reports (vs 67% for ChatGPT, P < .001). GPT-4 achieved higher accuracy in identification of metastatic disease (98.1% [95% CI: 97.7, 98.5] vs 90.3% [95% CI: 89.4, 91.0]) and higher performance in generating correct labels for oncologic progression (F1 score, 0.96 [95% CI: 0.94, 0.98] vs 0.91 [95% CI: 0.89, 0.94]) (both P < .001). In oncologic reasoning, GPT-4 had higher Likert scale scores for factual correctness (4.3 vs 3.9) and accuracy (4.4 vs 3.3), with a lower rate of confabulation (1.7% vs 13.7%) than ChatGPT (all P < .001). Conclusion When using user-defined prompts, GPT-4 outperformed ChatGPT in extracting oncologic phenotypes from free-text CT reports on lung cancer and demonstrated better oncologic reasoning with fewer confabulations. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Hafezi-Nejad and Trivedi in this issue.

Natural language processing to identify social determinants of health in Alzheimer's disease and related dementia from electronic health records.

To develop a natural language processing (NLP) algorithm that identifies social determinants of health (SDoH), including housing, transportation, food, and medication insecurities, social isolation, abuse, neglect, or exploitation, and financial difficulties for patients with Alzheimer's disease and related dementias (ADRD) from unstructured electronic health records (EHRs). We leveraged 1000 medical notes randomly selected from 7401 emergency department and inpatient social worker notes generated between 2015 and 2019 for 231 unique patients diagnosed with ADRD at Michigan Medicine. We developed a rule-based NLP algorithm for the identification of seven domains of SDoH noted above. We also compared the rule-based algorithm with deep learning and regularized logistic regression approaches. These models were compared using accuracy, sensitivity, specificity, F1 score, and the area under the receiver operating characteristic curve (AUC). All notes were split into 700 notes for training NLP algorithms, and 300 notes for validation. Social worker notes used in this study were extracted from the Michigan Medicine EHR database. Of the 700 notes for training, F1 and AUC for the rule-based algorithm were at least 0.94 and 0.95, respectively, for all SDoH categories. Of the 300 notes for validation, F1 and AUC were at least 0.80 and 0.97, respectively, for all SDoH except housing and medication insecurities. The deep learning and regularized logistic regression algorithms had unsatisfactory performance. The rule-based algorithm can accurately extract SDoH information in all seven domains of SDoH except housing and medication insecurities. Findings from the algorithm can be used by clinicians and social workers to proactively address social needs of patients with ADRD and other vulnerable patient populations.

Large-scale identification of undiagnosed hepatic steatosis using natural language processing

Nonalcoholic fatty liver disease (NAFLD) is a major cause of liver-related morbidity in people with and without diabetes, but it is underdiagnosed, posing challenges for research and clinical management. Here, we determine if natural language processing (NLP) of data in the electronic health record (EHR) could identify undiagnosed patients with hepatic steatosis based on pathology and radiology reports. A rule-based NLP algorithm was built using a Linguamatics literature text mining tool to search 2.15 million pathology report and 2.7 million imaging reports in the Penn Medicine EHR from November 2014, through December 2020, for evidence of hepatic steatosis. For quality control, two independent physicians manually reviewed randomly chosen biopsy and imaging reports (n=353, PPV 99.7%). After exclusion of individuals with other causes of hepatic steatosis, 3007 patients with biopsy-proven NAFLD and 42,083 patients with imaging-proven NAFLD were identified. Interestingly, elevated ALT was not a sensitive predictor of the presence of steatosis, and only half of the biopsied patients with steatosis ever received an ICD diagnosis code for the presence of NAFLD/NASH. There was a robust association for PNPLA3 and TM6SF2 risk alleles and steatosis identified by NLP. We identified 234 disorders that were significantly over- or underrepresented in all subjects with steatosis and identified changes in serum markers (e.g., GGT) associated with presence of steatosis. This study demonstrates clear feasibility of NLP-based approaches to identify patients whose steatosis was indicated in imaging and pathology reports within a large healthcare system and uncovers undercoding of NAFLD in the general population. Identification of patients at risk could link them to improved care and outcomes. The study was funded by US and German funding sources that did provide financial support only and had no influence or control over the research process.

RegEMR: a natural language processing system to automatically identify premature ovarian decline from Chinese electronic medical records

BackgroundThe ovarian reserve is a reservoir for reproductive potential. In clinical practice, early detection and treatment of premature ovarian decline characterized by abnormal ovarian reserve tests is regarded as a critical measure to prevent infertility. However, the relevant data are typically stored in an unstructured format in a hospital’s electronic medical record (EMR) system, and their retrieval requires tedious manual abstraction by domain experts. Computational tools are therefore needed to reduce the workload.MethodsWe presented RegEMR, an artificial intelligence tool composed of a rule-based natural language processing (NLP) extractor and a knowledge-based disease scoring model, to automatize the screening procedure of premature ovarian decline using Chinese reproductive EMRs. We used regular expressions (REs) as a text mining method and explored whether REs automatically synthesized by the genetic programming-based online platform RegexGenerator + + could be as effective as manually formulated REs. We also investigated how the representativeness of the learning corpus affected the performance of machine-generated REs. Additionally, we translated the clinical diagnostic criteria into a programmable disease diagnostic model for disease scoring and risk stratification. Four hundred outpatient medical records were collected from a Chinese fertility center. Manual review served as the gold standard, and fivefold cross-validation was used for evaluation.ResultsThe overall F-score of manually built REs was 0.9444 (95% CI 0.9373 to 0.9515), with no significant difference (paired t test p > 0.05) compared with machine-generated REs that could be affected by training set sizes and annotation portions. The extractor performed effectively in automatically tracing the dynamic changes in hormone levels (F-score 0.9518–0.9884) and ultrasonographic measures (F-score 0.9472–0.9822). Applying the extracted information to the proposed diagnostic model, the program obtained an accuracy of 0.98 and a sensitivity of 0.93 in risk screening. For each specific disease, the automatic diagnosis in 76% of patients was consistent with that of the clinical diagnosis, and the kappa coefficient was 0.63.ConclusionA Chinese NLP system named RegEMR was developed to automatically identify high risk of early ovarian aging and diagnose related diseases from Chinese reproductive EMRs. We hope that this system can aid EMR-based data collection and clinical decision support in fertility centers.

1287-P: Identifying When Incident Diabetes Was Diagnosed in Children and Young Adults, Using Natural Language Processing of Clinical Notes

Identifying when an incident diabetes (DM) diagnosis was made is complicated using retrospective, structured electronic health record (EHR) data alone. Unstructured clinical notes have been underused but contain valuable information that could complement traditional methods. However, manually reviewing clinical notes is time-consuming. We developed and validated a simple rule-based Natural Language Processing (NLP) method to extract incident DM timing from clinical notes. In a single center we used structured EHR data to identify a cohort (age &amp;lt;45 as of 12/31/19) with likely type 1 (T1D) or type 2 (T2D) DM based on 2016-2019 records: (≥1 T1D ICD-10 code and insulin and no other DM medication) or ([≥2 T2D and no T1D codes] or [≥1 T2D code and a DM medication besides insulin or metformin]). This cohort had 2,654 patients (548,316 clinical notes, 2003-present). We randomly selected 58,450 clinical notes (1,465 patients) as a training set to look for relevant text patterns. We handcrafted the rules into our NLP tool. We required 3 distinct concepts at the sentence level to determine an incident DM diagnosis: DM (not, e.g., epilepsy), an onset attribute (e.g., “diagnosed in”), and a temporal component (e.g., 8/2008). We pre-defined all related keywords and date formats for these concepts in our training notes. We then tested the NLP algorithm against manual review in an independent set of 100 randomly selected patients from the cohort. Analysis was at the patient level (true+: ≥1 true+ note per patient). NLP in the training set found 1,268 patients with at least 1 of the 3 concepts and 826 patients with all 3. In the test set, we excluded 4 patients without substantive notes. NLP correctly detected incident DM timing in 73 of 96 patients. The NLP had recall 88%, specificity 77%, precision (PPV) 96%, and NPV 50%. NLP was helpful in finding incident DM timing and may complement structured EHR queries for identifying incident DM. Refinement of our NLP algorithm is ongoing. Disclosure A.Wong: None. V.W.Zhong: None. M.Rosenman: None. Funding Centers for Disease Control and Prevention (1U18DP006693-01-00)