Open-source Natural Language Processing Research Articles

ChatGPT vs. neurologists: a cross-sectional study investigating preference, satisfaction ratings and perceived empathy in responses among people living with multiple sclerosis.

ChatGPT is an open-source natural language processing software that replies to users' queries. We conducted a cross-sectional study to assess people living with Multiple Sclerosis' (PwMS) preferences, satisfaction, and empathy toward two alternate responses to four frequently-asked questions, one authored by a group of neurologists, the other by ChatGPT. An online form was sent through digital communication platforms. PwMS were blind to the author of each response and were asked to express their preference for each alternate response to the four questions. The overall satisfaction was assessed using a Likert scale (1-5); the Consultation and Relational Empathy scale was employed to assess perceived empathy. We included 1133 PwMS (age, 45.26 ± 11.50years; females, 68.49%). ChatGPT's responses showed significantly higher empathy scores (Coeff = 1.38; 95% CI = 0.65, 2.11; p > z < 0.01), when compared with neurologists' responses. No association was found between ChatGPT' responses and mean satisfaction (Coeff = 0.03; 95% CI = -0.01, 0.07; p = 0.157). College graduate, when compared with high school education responder, had significantly lower likelihood to prefer ChatGPT response (IRR = 0.87; 95% CI = 0.79, 0.95; p < 0.01). ChatGPT-authored responses provided higher empathy than neurologists. Although AI holds potential, physicians should prepare to interact with increasingly digitized patients and guide them on responsible AI use. Future development should consider tailoring AIs' responses to individual characteristics. Within the progressive digitalization of the population, ChatGPT could emerge as a helpful support in healthcare management rather than an alternative.

Natural Language Processing in Knowledge-Based Support for Operator Assistance

Manufacturing industry faces increasing complexity in the performance of assembly tasks due to escalating demand for complex products with a greater number of variations. Operators require robust assistance systems to enhance productivity, efficiency, and safety. However, existing support services often fall short when operators encounter unstructured open questions and incomplete sentences due to primarily relying on procedural digital work instructions. This draws attention to the need for practical application of natural language processing (NLP) techniques. This study addresses these challenges by introducing a domain-specific dataset tailored to assembly tasks, capturing unique language patterns and linguistic characteristics. We explore strategies to process declarative and imperative sentences, including incomplete ones, effectively. Thorough evaluation of three pre-trained NLP libraries—NLTK, SPACY, and Stanford—is performed to assess their effectiveness in handling assembly-related concepts and ability to address the domain’s distinctive challenges. Our findings demonstrate the efficient performance of these open-source NLP libraries in accurately handling assembly-related concepts. By providing valuable insights, our research contributes to developing intelligent operator assistance systems, bridging the gap between NLP techniques and the assembly domain within manufacturing industry.

Alignment of Unsupervised Machine Learning with Human Understanding: A Case Study of Connected Vehicle Patents

As official public records of inventions, patents provide an understanding of technological trends across the competitive landscape of various industries. However, traditional manual analysis methods have become increasingly inadequate due to the rapid expansion of patent information and its unstructured nature. This paper contributes an original approach to enhance the understanding of patent data, with connected vehicle (CV) patents serving as the case study. Using free, open-source natural language processing (NLP) libraries, the author introduces a novel metric to quantify the alignment of classifications by a subject matter expert (SME) and using machine learning (ML) methods. The metric is a composite index that includes a purity factor, evaluating the average ML conformity across SME classifications, and a dispersion factor, assessing the distribution of ML assigned topics across these classifications. This dual-factor approach, labeled the H-index, quantifies the alignment of ML models with SME understanding in the range of zero to unity. The workflow utilizes an exhaustive combination of state-of-the-art tokenizers, normalizers, vectorizers, and topic modelers to identify the best NLP pipeline for ML model optimization. The study offers manifold visualizations to provide an intuitive understanding of the areas where ML models align or diverge from SME classifications. The H-indices reveal that although ML models demonstrate considerable promise in patent analysis, the need for further advancements remain, especially in the domain of patent analysis.

Making Use of Natural Language Processing to Better Understand Medical Students' Self-Assessment of Clinical Skills.

Reflective practice is necessary for self-regulated learning. Helping medical students develop these skills can be challenging since they are difficult to observe. One common solution is to assign students' reflective self-assessments, which produce large quantities of narrative assessment data. Reflective self-assessments also provide feedback to faculty regarding students' understanding of content, reflective abilities, and areas for course improvement. To maximize student learning and feedback to faculty, reflective self-assessments must be reviewed and analyzed, activities that are often difficult for faculty due to the time-intensive and cumbersome nature of processing large quantities of narrative assessment data. The authors collected narrative assessment data (2,224 students' reflective self-assessments) from 344 medical students' reflective self-assessments. In academic years 2019-2020 and 2021-2022, students at the University of Cincinnati College of Medicine responded to 2 prompts (aspects that surprised students, areas for student improvement) after reviewing their standardized patient encounters. These free-text entries were analyzed using TopEx, an open-source natural language processing (NLP) tool, to identify common topics and themes, which faculty then reviewed. TopEx expedited theme identification in students' reflective self-assessments, unveiling 10 themes for prompt 1 such as question organization and history analysis, and 8 for prompt 2, including sensitive histories and exam efficiency. Using TopEx offered a user-friendly, time-saving analysis method without requiring complex NLP implementations. The authors discerned 4 education enhancement implications: aggregating themes for future student reflection, revising self-assessments for common improvement areas, adjusting curriculum to guide students better, and aiding faculty in providing targeted upcoming feedback. The University of Cincinnati College of Medicine aims to refine and expand the utilization of TopEx for deeper narrative assessment analysis, while other institutions may model or extend this approach to uncover broader educational insights and drive curricular advancements.

An open-source natural language processing toolkit to support software development: addressing automatic bug detection, code summarisation and code search.

This paper aims to introduce the innovative work carried out in the Horizon 2020 DECODER project - acronym for "DEveloper COmpanion for Documented and annotatEd code Reference" - (Grant Agreement no. 824231) by linking the fields of natural language processing (NLP) and software engineering. The project as a whole addresses the development of a framework, namely the Persistent Knowledge Monitor (PKM), that acts as a central infrastructure to store, access, and trace all the data, information and knowledge related to a given software or ecosystem. This meta-model defines the knowledge base that can be queried and analysed by all the tools integrated and developed in DECODER. Besides, the DECODER project offers a friendly user interface where each of the predefined three roles (i.e., developers, maintainers and reviewers) can access and query the PKM with their personal accounts. The paper focuses on the NLP tools developed and integrated in the PKM, namely the deep learning models developed to perform variable misuse, code summarisation and semantic parsing. These were developed under a common work package - "Activities for the developer" - intended to precisely target developers, who can perform tasks such as detection of bugs, automatic generation of documentation for source code and generation of code snippets from natural languages instructions, among the multiple functionalities that DECODER offers. These tools assist and help the developers in the daily work, by increasing their productivity and avoiding loss of time in tedious tasks such as manual bug detection. Training and validation were conducted for four use cases in Java, C and C++ programming languages in order to evaluate the performance, suitability, usability, etc. of the developed tools.

Natural Language Processing for the Ascertainment and Phenotyping of Left Ventricular Hypertrophy and Hypertrophic Cardiomyopathy on Echocardiogram Reports

Extracting and accurately phenotyping electronic health documentation is critical for medical research and clinical care. We sought to develop a highly accurate and open-source natural language processing (NLP) module to ascertain and phenotype left ventricular hypertrophy (LVH) and hypertrophic cardiomyopathy (HCM) diagnoses fromechocardiogram reports within a diverse hospital network. After the initial development on 17,250 echocardiogram reports, 700 unique reports from 6 hospitals were randomly selected from data repositories within the Mass General Brigham healthcare system and manually adjudicated by physicians for 10 subtypes of LVH and diagnoses of HCM. Using an open-source NLP system, the module was formally tested on 300 training set reports and validated on 400 reports. The sensitivity, specificity, positive predictive value, and negative predictive value were calculated to assess the discriminative accuracy of the NLP module. The NLP demonstrated robust performance across the 10 LVH subtypes, with the overall sensitivity and specificity exceeding 96%. In addition, the NLP module demonstrated excellent performance in detecting HCM diagnoses, with sensitivity and specificity exceeding 93%. In conclusion, we designed a highly accurate NLP module to determine the presence of LVH and HCM on echocardiogram reports. Our work demonstrates the feasibility and accuracy of NLP to detect diagnoses on imaging reports, even when described in free text. This module has been placed in the public domain to advance research, trial recruitment, and population health management for patients with LVH-associated conditions.

The Subgroup Imperative: Chest Radiograph Classifier Generalization Gaps in Patient, Setting, and Pathology Subgroups.

To externally test four chest radiograph classifiers on a large, diverse, real-world dataset with robust subgroup analysis. In this retrospective study, adult posteroanterior chest radiographs (January 2016-December 2020) and associated radiology reports from Trillium Health Partners in Ontario, Canada, were extracted and de-identified. An open-source natural language processing tool was locally validated and used to generate ground truth labels for the 197 540-image dataset based on the associated radiology report. Four classifiers generated predictions on each chest radiograph. Performance was evaluated using accuracy, positive predictive value, negative predictive value, sensitivity, specificity, F1 score, and Matthews correlation coefficient for the overall dataset and for patient, setting, and pathology subgroups. Classifiers demonstrated 68%-77% accuracy, 64%-75% sensitivity, and 82%-94% specificity on the external testing dataset. Algorithms showed decreased sensitivity for solitary findings (43%-65%), patients younger than 40 years (27%-39%), and patients in the emergency department (38%-60%) and decreased specificity on normal chest radiographs with support devices (59%-85%). Differences in sex and ancestry represented movements along an algorithm's receiver operating characteristic curve. Performance of deep learning chest radiograph classifiers was subject to patient, setting, and pathology factors, demonstrating that subgroup analysis is necessary to inform implementation and monitor ongoing performance to ensure optimal quality, safety, and equity.Keywords: Conventional Radiography, Thorax, Ethics, Supervised Learning, Convolutional Neural Network (CNN), Machine Learning Algorithms Supplemental material is available for this article. © RSNA, 2023See also the commentary by Huisman and Hannink in this issue.

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults.

The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS. ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.

An evaluation of existing text de-identification tools for use with patient progress notes from Australian general practice

IntroductionDigitized patient progress notes from general practice represent a significant resource for clinical and public health research but cannot feasibly and ethically be used for these purposes without automated de-identification. Internationally, several open-source natural language processing tools have been developed, however, given wide variations in clinical documentation practices, these cannot be utilized without appropriate review. We evaluated the performance of four de-identification tools and assessed their suitability for customization to Australian general practice progress notes. MethodsFour tools were selected: three rule-based (HMS Scrubber, MIT De-id, Philter) and one machine learning (MIST). 300 patient progress notes from three general practice clinics were manually annotated with personally identifying information. We conducted a pairwise comparison between the manual annotations and patient identifiers automatically detected by each tool, measuring recall (sensitivity), precision (positive predictive value), f1-score (harmonic mean of precision and recall), and f2-score (weighs recall 2x higher than precision). Error analysis was also conducted to better understand each tool’s structure and performance. ResultsManual annotation detected 701 identifiers in seven categories. The rule-based tools detected identifiers in six categories and MIST in three. Philter achieved the highest aggregate recall (67%) and the highest recall for NAME (87%). HMS Scrubber achieved the highest recall for DATE (94%) and all tools performed poorly on LOCATION. MIST achieved the highest precision for NAME and DATE while also achieving similar recall to the rule-based tools for DATE and highest recall for LOCATION. Philter had the lowest aggregate precision (37%), however preliminary adjustments of its rules and dictionaries showed a substantial reduction in false positives. ConclusionExisting off-the-shelf solutions for automated de-identification of clinical text are not immediately suitable for our context without modification. Philter is the most promising candidate due to its high recall and flexibility however will require extensive revising of its pattern matching rules and dictionaries.

Topic modeling and sentiment analysis of Chinese people’s attitudes toward volunteerism amid the COVID-19 pandemic

The COVID-19 pandemic has created an urgent need for volunteers to complement overwhelmed public health systems. This study aims to explore Chinese people's attitudes toward volunteerism amid the COVID-19 pandemic. To this end, we identify the latent topics in volunteerism-related microblogs on Weibo, the Chinese equivalent of Twitter using the topic modeling analysis via Latent Dirichlet Allocation (LDA). To further investigate the public sentiment toward the topics generated by LDA, we also conducted sentiment analysis on the sample posts using the open-source natural language processing (NLP) technique from Baidu. Through an in-depth analysis of 91,933 Weibo posts, this study captures 10 topics that are, in turn, distributed into five factors associated with volunteerism in China as motive fulfillment (n = 31,661, 34.44%), fear of COVID-19 (n = 22,597, 24.58%), individual characteristic (n = 17,688, 19.24%), government support (n = 15,482, 16.84%), and community effect (n = 4,505, 4.90%). The results show that motive fulfillment, government support, and community effect are the factors that could enhance positive attitudes toward volunteerism since the topics related to these factors report high proportions of positive emotion. Fear of COVID-19 and individual characteristic are the factors inducing negative sentiment toward volunteerism as the topics related to these factors show relatively high proportions of negative emotion. The provision of tailored strategies based on the factors could potentially enhance Chinese people's willingness to participate in volunteer activities during the COVID-19 pandemic.

Integration of Augmented Reality and Voice Recognition in Learning English for Children

Application development by combining two technologies, namely Augmented Reality and Voice Recognition, can make learning media regarding object recognition at home interact directly with 3D virtual objects. The technology can also help the pronunciation or pronunciation of sentences in English. Natural Language Processing (NLP) is used to understand human language so that machines can understand and process it. This ability supports Voice Recognition to have intelligence and interact like humans. wit.ai is an open-source NLP platform that can support speech-to-text application development. The merging of the two technologies in this development using the wit.ai platform. With the wit.ai platform that is used to understand voice commands and perform tasks as needed for applications regarding object recognition at home, users will be able to interact with objects at home through the given voice commands. In the Black Box testing, each functionality got the results that all the features had functioned properly. User Acceptance Test was also carried out and the average test results were 95.77% and 93.26% on a Likert scale with test results on 13 respondents aged 6-9 years old who have tried the application and 14 respondents as observers when 13 respondents aged 6-9 years tried the application. These results show that the application can be accepted and used as a tool in learning media.

Utilization of Natural Language Processing Software to Identify Worrisome Pancreatic Lesions.

Computed tomography (CT) imaging is routinely obtained for diagnostics, especially in trauma and emergency rooms, often identifying incidental findings. We utilized a natural language processing (NLP) algorithm to quantify the incidence of clinically relevant pancreatic lesions in CT imaging. We utilized the electronic medical record to perform a retrospective chart review of adult patients admitted for trauma to a level 1 tertiary care center between 2010 and 2020 who underwent abdominal CT imaging. An open-source NLP software was used to identify patients with intrapapillary mucinous neoplasms (IPMN), pancreatic cysts, pancreatic ductal dilation, or pancreatic masses after optimizing the algorithm using a test group of patients who underwent pancreatic surgery. The algorithm identified pancreatic lesions in 27 of 28 patients who underwent pancreatic surgery and excluded 1 patient who had a pure ampullary mass. The study cohort consisted of 18,769 patients who met our inclusion criteria admitted to the hospital. Of this population, 232 were found to have pancreatic lesions of interest. There were 48 (20.7%) patients with concern for IPMN, pancreatic cysts in 36 (15.5%), concerning masses in 30 (12.9%), traumatic findings in 44 (19.0%), pancreatitis in 41 (17.7%), and ductal abnormalities in 19 (18.2%) patients. Prior pancreatic surgery and other findings were identified in 14 (6.0%) patients. In this study, we propose a novel use of NLP software to identify potentially malignant pancreatic lesions annotated in CT imaging performed for other purposes. This methodology can significantly increase the screening and automated referral for the management of precancerous lesions.

An open-source natural language processing toolkit to support software development: addressing automatic bug detection, code summarisation and code search

This paper aims to introduce the innovative work carried out in the Horizon 2020 DECODER project – acronym for “DEveloper COmpanion for Documented and annotatEd code Reference” – (Grant Agreement no. 824231) by linking the fields of natural language processing (NLP) and software engineering. The project as a whole addresses the development of a framework, namely the Persistent Knowledge Monitor (PKM), that acts as a central infrastructure to store, access, and trace all the data, information and knowledge related to a given software or ecosystem. This meta-model defines the knowledge base that can be queried and analysed by all the tools integrated and developed in DECODER. Besides, the DECODER project offers a friendly user interface where each of the predefined three roles (i.e., developers, maintainers and reviewers) can access and query the PKM with their personal accounts. The paper focuses on the NLP tools developed and integrated in the PKM, namely the deep learning models developed to perform variable misuse, code summarisation and semantic parsing. These were developed under a common work package – “Activities for the developer” – intended to precisely target developers, who can perform tasks such as detection of bugs, automatic generation of documentation for source code and generation of code snippets from natural languages instructions, among the multiple functionalities that DECODER offers. These tools assist and help the developers in the daily work, by increasing their productivity and avoiding loss of time in tedious tasks such as manual bug detection. Training and validation were conducted for four use cases in Java, C and C++ programming languages in order to evaluate the performance, suitability, usability, etc. of the developed tools.

Investigation of the Utility of Features in a Clinical De-identification Model: A Demonstration Using EHR Pathology Reports for Advanced NSCLC Patients.

BackgroundElectronic health record (EHR) systems contain a large volume of texts, including visit notes, discharge summaries, and various reports. To protect the confidentiality of patients, these records often need to be fully de-identified before circulating for secondary use. Machine learning (ML) based named entity recognition (NER) model has emerged as a popular technique of automatic de-identification.ObjectiveThe performance of a machine learning model highly depends on the selection of appropriate features. The objective of this study was to investigate the usability of multiple features in building a conditional random field (CRF) based clinical de-identification NER model.MethodsUsing open-source natural language processing (NLP) toolkits, we annotated protected health information (PHI) in 1,500 pathology reports and built supervised NER models using multiple features and their combinations. We further investigated the dependency of a model's performance on the size of training data.ResultsAmong the 10 feature extractors explored in this study, n-gram, prefix–suffix, word embedding, and word shape performed the best. A model using combination of these four feature sets yielded precision, recall, and F1-score for each PHI as follows: NAME (0.80; 0.79; 0.80), LOCATION (0.85; 0.83; 0.84), DATE (0.86; 0.79; 0.82), HOSPITAL (0.96; 0.93; 0.95), ID (0.99; 0.82; 0.90), and INITIALS (0.97; 0.49; 0.65). We also found that the model's performance becomes saturated when the training data size is beyond 200.ConclusionManual de-identification of large-scale data is an impractical procedure since it is time-consuming and subject to human errors. Analysis of the NER model's performance in this study sheds light on a semi-automatic clinical de-identification pipeline for enterprise-wide data warehousing.

Abstract 9389: Apache Clinical Text Analysis Knowledge Extraction System (ctakes) Identifies One Out of Five Suspicious Symptoms of Acute Coronary Syndrome from Emergency Department Nursing Triage Notes

Introduction: Acute coronary syndrome is a time sensitive condition requiring rapid identification, but nurses only identify 40%-50% of high-risk patients at emergency department (ED) triage. Symptoms are the driving force behind prompt ED care of patients with ACS. Symptom data are recorded as free text in nursing narratives. Standardized extraction of symptoms via natural language processing will create usable clinical data that could be used in predictive analytics, to improve early identification of ACS and ensure life-saving treatments. Hypothesis: The Apache Clinical Text Analysis and Knowledge Extraction System (cTAKES) will recognize symptom concepts in nursing narratives with an acceptable F1 score of greater than or equal to 0.9. Methods: A nurse scientist and a physician annotated 1200 ED nursing narratives of patients with suspicion of ACS for the following symptoms predictive of ACS: chest pain, chest discomfort, shortness of breath, fatigue and sweating. Inter-rater reliability (Cohen’s kappa statistic) was calculated with 10% overlap. cTAKES, which is an open-source natural language processing platform, was applied to extract evidence of the 5 annotated symptoms. Recall (sensitivity), precision (positive predictive value) and F1 score (harmonic mean of precision and recall) were calculated to assess extraction quality for each symptom. Results: Recall, precision and F1 score for each symptom are as follows: chest pain: 0.71, 0.97, 0.823; chest discomfort: 0.6, 0.9, 0.72; fatigue: 0.9, 0.26, 0.41; shortness of breath: 0.93, 0.96, 0.95; and sweating: 0.54, 1, 0.7, respectively. Cohen’s kappa was 0.9. Conclusions: cTAKES had fair results across all 5 ACS predictive symptoms, with shortness of breath meeting our hypothesized F1 score goal. Failure to recognize symptoms at a clinically acceptable value might be due to the linguistic and content variation present in ED nursing initial narratives. Customized natural language processing pipelines trained specifically for ED nursing narratives have significant potential for accurately identifying symptoms and improving early identification of ACS.

Derivation and Validation of Natural Language Processing Algorithms to Identify and Classify Venous Thrombotic Events from Lower Extremity Duplex Ultrasound Reports

Background: The precise anatomic location and extent of most venous thromboembolism (VTE) is not captured through diagnostic codes and is a major limitation to research. Correctly identifying the specific site of thrombosis, whether proximal or distal deep vein thrombosis (DVT) or superficial vein thrombosis (SVT) is critical.An accurate natural language processing (NLP) tool would make analysis of large datasets from electronic medical records possible and could be an significant improvement compared to analyses using the International Classification of Diseases codes. Using an open-source NLP tool, we evaluated existing algorithms and then created and validated new NLP algorithms to classify lower extremity thrombosis as not only a DVT or SVT, but also to identify more precise anatomic locations.Methods: A random sample of deidentified ultrasound reports were extracted from electronic medical records, manually reviewed, and classified as either positive or negative for a DVT or SVT (any chronicity). Reports with DVT were further classified into proximal (popliteal, femoral, deep femoral, common femoral, iliac veins, or vena cava) and distal (posterior tibial, anterior tibial, peroneal, soleal, or gastrocnemius veins) DVT. SVT was further classified into greater saphenous vein, small saphenous vein, or other site. Thrombosis near the junction of deep and superficial veins was only designated as involving specific vein segments if there was intraluminal thrombosis at that site. Initial sets of 100 ultrasound reports were used for derivation and reiterative testing of the algorithm, consisting of 50 that were positive for DVT/SVT and 50 that were negative for DVT/SVT. Text from radiology reports underwent cleaning to remove extraneous punctuation, text outside of the “findings” and “impression” section, and all text matches for “superficial femoral vein” were replaced with “femoral vein”. Using target phrases, the simple NLP tool either classified the reports as present or absent, signifying either positive or negative for thrombosis at the site of interest. After maximizing accuracy in the derivation cohorts, each NLP algorithm was tested in the complete, manually reviewed dataset. Sensitivity (Sn) and specificity (Sp) were calculated and confidence intervals were determined by the binomial exact method.Results: A total of 1206 ultrasounds were reviewed, among which there was 687 positives for DVT (503 had proximal DVT and 378 had distal DVT involvement). A total of 176 had SVT, 114 with involvement of the great saphenous vein (GSV) and 65 with involvement of the small saphenous vein (SSV). The previously published NLP (designed to identify DVT) had a poor Sn 45.0%, but reasonable Sp (91.0%) for DVT. Among the incorrect positive determinations (n=47), 33 were incorrectly positive due to presence of SVT (in the absence of DVT). Our newly developed NLP algorithm correctly identified DVT at any site (Sn: 96.2% 95% CI 94.5-97.5; 661/687 and Sp: 93.8% 95% CI 91.4-95.7; 487/519). Among ultrasounds positive for DVT, the NLP algorithm to determine proximal DVT site had a 96.0% Sn (95% CI 93.9-97.6; 483/503) and a 97.8% Sp (95% CI 94.5-99.4; 179/183). The algorithm for distal DVT had a 96.8% Sn (95% CI 94.5-98.4; 366/378) and a 96.1% Sp (95% CI 93.3-98.0; 296/308). The algorithm to identify SVT was also highly accurate (Sn 97.7%, 95% CI 94.3-99.4; 172/176 and Sp 95.5%, 95% CI 94.1-96.7; 984/1030). Among ultrasounds positive for SVT, the Sn to determine GSV site was 94.7% (95% CI 88.9-98.0; 108/114) and Sp was 95.1% (95% CI 86.3-99.0; 58/61). The algorithm to identify SSV site had a Sn of 98.5% (95% CI 91.7-99.9; 64/65) and a Sp of 100% (95% CI 96.7-100.0; 110/110).Conclusion: Using a previously published, open source, simple NLP program, we have improved the sensitivity and specificity for identification of DVT and created an algorithm to accurately identify SVT. Using a multifaceted analysis approach, we were able to accurately further subclassify the anatomic location of thrombosis on ultrasound reports. This tool and the developed algorithms will allow analysis of large data sets with minimal effort and great accuracy, capitalizing on the power of large electronic datasets to offer new insights on pathophysiology and clinical prognosis. [Display omitted] DisclosuresNo relevant conflicts of interest to declare.

NLP-based personal learning assistant for school education

Computer-based knowledge and computation systems are becoming major sources of leverage for multiple industry segments. Hence, educational systems and learning processes across the world are on the cusp of a major digital transformation. This paper seeks to explore the concept of an artificial intelligence and natural language processing (NLP) based intelligent tutoring system (ITS) in the context of computer education in primary and secondary schools. One of the components of an ITS is a learning assistant, which can enable students to seek assistance as and when they need, wherever they are. As part of this research, a pilot prototype chatbot was developed, to serve as a learning assistant for the subject Scratch (Scratch is a graphical utility used to teach school children the concepts of programming). By the use of an open source natural language understanding (NLU) or NLP library, and a slackbased UI, student queries were input to the chatbot, to get the sought explanation as the answer. Through a two-stage testing process, the chatbot’s NLP extraction and information retrieval performance were evaluated. The testing results showed that the ontology modelling for such a learning assistant was done relatively accurately, and shows its potential to be pursued as a cloud-based solution in future.

Visualizing NLP in Undergraduate Students' Learning about Natural Language

We report on the use of open-source natural language processing capabilities in a web-based interface to allow undergraduate students to apply what they have learned about formal natural language structures. The learning activities encourage students to interpret data in new ways, think originally about natural language, and critique the back-end NLP models and algorithms visualized on the user front end. This work is of relevance to AI resources developed for education by focusing on inclusivity of students from many disciplinary backgrounds. Specifically, we comprehensively extended a web-based system with new resources. To test the students' reactions to NLP analyses that offer insights into both the strengths and limitations of AI systems, we incorporated a range of automated analyses focused on language-independent processing or meaning representations which still represent challenges for NLP. We conducted a survey-based evaluation with students in open-ended case-based assignments in undergraduate coursework. Responses indicated that the students reinforced their knowledge, applied critical thinking about language and NLP applications, and used the application not to solve the assignment for them, but as a tool in their own effort to address the task. We further discuss how using interpretable visualizations of system decisions is an opportunity to learn about ethical issues in NLP, and how making AI systems interpretable may broaden multidisciplinary interest in AI in early educational experiences.

Automated analysis of ambulatory surgery patient experience comments using artificial intelligence for quality improvement: A patient centered approach

BackgroundQuality improvement in healthcare is limited by both the quality and quantity of data available in the electronic health records or self reported by clinicians. Appropriate and timely reporting can help identify opportunities for quality improvement and deployment of mitigation steps but many of these process remain manual. Negative patient experience comments can potentially be used in identifying opportunities for quality improvement. Artificial intelligence (AI) techniques, like natural language processing (NLP) including sentiment analysis and topic modeling can be used to characterize patient experience comments in near real time, thus making the process timely, efficient and measurable. MethodsWe analyzed 15,453 de-identified patient experience comments from the Press Ganey™ survey of adult patients undergoing outpatient surgeries with anesthesia at Cleveland Clinic from 01/01/2012-05/03/2016. We used open source NLP including sentiment analysis and topical modeling to analyze post-discharge patient experience survey comments and feedback verbatim. For sentiment analysis we used an open source sentiment analyzer called VADER. For topic modeling we used Latent Dirichlet Allocation (LDA) algorithm. ResultsSentiment analysis of patient comments using VADER was highly accurate with F1 score ranging from 0.83 to 0.84 for positive comments and 0.71–0.73 for negative comments compared to clinician's assessment. Two clinicians reviewed 1955 random comments as positive or negative with good agreement, with Cohen's k test score of k = 0.92 (p < 0.001). Six thematic areas were identified based on LDA algorithms topic analysis with interpretable and measurable weighable of words extracted from the comments associated with poor patient satisfaction, thus defining targeted opportunities for improvement. ConclusionWe conclude that artificial intelligence can help in near real-time analysis of patient experience surveys. Commonly used open source algorithms can possibly be utilized in healthcare for quality improvement negating the need for significant development resources with the opportunity to generalize and scale.

Identifying Symptom Information in Clinical Notes Using Natural Language Processing.

Symptoms are a core concept of nursing interest. Large-scale secondary data reuse of notes in electronic health records (EHRs) has the potential to increase the quantity and quality of symptom research. However, the symptom language used in clinical notes is complex. A need exists for methods designed specifically to identify and study symptom information from EHR notes. We aim to describe a method that combines standardized vocabularies, clinical expertise, and natural language processing to generate comprehensive symptom vocabularies and identify symptom information in EHR notes. We piloted this method with five diverse symptom concepts: constipation, depressed mood, disturbed sleep, fatigue, and palpitations. First, we obtained synonym lists for each pilot symptom concept from the Unified Medical Language System. Then, we used two large bodies of text (clinical notes from Columbia University Irving Medical Center and PubMed abstracts containing Medical Subject Headings or key words related to the pilot symptoms) to further expand our initial vocabulary of synonyms for each pilot symptom concept. We used NimbleMiner, an open-source natural language processing tool, to accomplish these tasks and evaluated NimbleMiner symptom identification performance by comparison to a manually annotated set of nurse- and physician-authored common EHR note types. Compared to the baseline Unified Medical Language System synonym lists, we identified up to 11 times more additional synonym words or expressions, including abbreviations, misspellings, and unique multiword combinations, for each symptom concept. Natural language processing system symptom identification performance was excellent. Using our comprehensive symptom vocabularies and NimbleMiner to label symptoms in clinical notes produced excellent performance metrics. The ability to extract symptom information from EHR notes in an accurate and scalable manner has the potential to greatly facilitate symptom science research.