NLP Pipeline Research Articles

Annotation of epilepsy clinic letters for natural language processing

BackgroundNatural language processing (NLP) is increasingly being used to extract structured information from unstructured text to assist clinical decision-making and aid healthcare research. The availability of expert-annotated documents for the development and validation of NLP applications is limited. We created synthetic clinical documents to address this, and to validate the Extraction of Epilepsy Clinical Text version 2 (ExECTv2) NLP pipeline.MethodsWe created 200 synthetic clinic letters based on hospital outpatient consultations with epilepsy specialists. The letters were double annotated by trained clinicians and researchers according to agreed guidelines. We used the annotation tool, Markup, with an epilepsy concept list based on the Unified Medical Language System ontology. All annotations were reviewed, and a gold standard set of annotations was agreed and used to validate the performance of ExECTv2.ResultsThe overall inter-annotator agreement (IAA) between the two sets of annotations produced a per item F1 score of 0.73. Validating ExECTv2 using the gold standard gave an overall F1 score of 0.87 per item, and 0.90 per letter.ConclusionThe synthetic letters, annotations, and annotation guidelines have been made freely available. To our knowledge, this is the first publicly available set of annotated epilepsy clinic letters and guidelines that can be used for NLP researchers with minimum epilepsy knowledge. The IAA results show that clinical text annotation tasks are difficult and require a gold standard to be arranged by researcher consensus. The results for ExECTv2, our automated epilepsy NLP pipeline, extracted detailed epilepsy information from unstructured epilepsy letters with more accuracy than human annotators, further confirming the utility of NLP for clinical and research applications.

Semantic Mapping of Named-Entities in openEHR Templates and Ad-hoc Generation of Compositions.

Integration of free texts from reports written by physicians to an interoperable standard is important for improving patient-centric care and research in the medical domain. In the context of unstructured clinical data, NLP Information Extraction serves in finding information in unstructured text. To our best knowledge, there is no efficient solution, in which extracted Named-Entities of an NLP pipeline can be ad-hoc inserted in openEHR compositions. We therefore developed a software solution that solves this data integration problem by mapping Named-Entities of an NLP pipeline to the fields of an openEHR template. The mapping can be accomplished by any user without any programming intervention and allows the ad-hoc creation of a composition based on the mappings.

From virus to viral: content analysis of HIV-related Twitter messages among young men in the U.S.

BackgroundAdolescents and young adults account for over 21% of new HIV infections in the U.S. with most new cases among young men. As an important information source for this group, social media can uniquely reveal the perspectives and communicative patterns of this key population. We identified 6,439 young male Twitter users (ages 13–24) in the U.S. using an NLP pipeline with geolocations. From their Twitter timelines, we collected 24,600 HIV-related tweets, among which the most retweeted and favorited tweets (n = 472) were analyzed through a content analysis.ResultsThree themes arose in this online viral discourse around HIV among young men: (i) othering, (ii) politics and activism, (iii) risk and wellness. Othering tweets contained stigmatizing jokes and insults alienating individuals who identify as lesbian, gay, bisexual, transgender, queer or questioning, intersex, asexual, or being elsewhere on the gender and sexuality spectrum (LGBTQIA +), and people with HIV. Politics and activism tweets discussed awareness, stigma, HIV criminalization, violence, LGBTQIA + , and women’s rights. Risk and wellness tweets discussed risk behaviors for sexually transmitted infections (STIs) (e.g., condomless sex, transactional sex, multiple sexual partners), or safer sex and preventive practices (e.g., pre-exposure prophylaxis [PrEP], condom use, achieving undetectable viral load, medication adherence, and STI testing).ConclusionThe social acceptability of high-risk sex behaviors is high among young male Twitter users. Given the double-edged nature of social media—health-promoting (e.g., awareness, health activism) as well as risk-promoting (e.g., risky behavior endorsement, identity attacks)— this population may benefit from targeted health communication intervention. Future HIV prevention efforts should counter the stigma, misinformation, and risk-promoting viral messages prevalent on social media.

Unveiling Fall Risk Factors: Nurse-Driven Corpus Development for Natural Language Processing.

Hospital-acquired falls are a continuing clinical concern. The emergence of advanced analytical methods, including NLP, has created opportunities to leverage nurse-generated data, such as clinical notes, to better address the problem of falls. In this nurse-driven study, we employed an iterative process for expert manual annotation of RNs clinical notes to enable the training and testing of an NLP pipeline to extract factors related to falls. The resulting annotated data corpus had moderately high interrater reliability (F-score=0.74) and captured a breadth of clinical concepts for extraction with potential utility beyond patient falls. Further research is needed to determine which annotation tasks most benefit from nursing expert annotators, to optimize efficiency when tapping into the invaluable resource represented by the nursing workforce.

ReDWINE: A clinical datamart with text analytical capabilities to facilitate rehabilitation research

Rehabilitation research focuses on determining the components of a treatment intervention, the mechanism of how these components lead to recovery and rehabilitation, and ultimately the optimal intervention strategies to maximize patients’ physical, psychologic, and social functioning. Traditional randomized clinical trials that study and establish new interventions face challenges, such as high cost and time commitment. Observational studies that use existing clinical data to observe the effect of an intervention have shown several advantages over RCTs. Electronic Health Records (EHRs) have become an increasingly important resource for conducting observational studies. To support these studies, we developed a clinical research datamart, called ReDWINE (Rehabilitation Datamart With Informatics iNfrastructure for rEsearch), that transforms the rehabilitation-related EHR data collected from the UPMC health care system to the Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) to facilitate rehabilitation research. The standardized EHR data stored in ReDWINE will further reduce the time and effort required by investigators to pool, harmonize, clean, and analyze data from multiple sources, leading to more robust and comprehensive research findings. ReDWINE also includes deployment of data visualization and data analytics tools to facilitate cohort definition and clinical data analysis. These include among others the Open Health Natural Language Processing (OHNLP) toolkit, a high-throughput NLP pipeline, to provide text analytical capabilities at scale in ReDWINE. Using this comprehensive representation of patient data in ReDWINE for rehabilitation research will facilitate real-world evidence for health interventions and outcomes.

LayoutQT—Layout Quadrant Tags to embed visual features for document analysis

The relative position of text blocks plays a crucial role in document understanding. However, the task of embedding layout information in the representation of a page instance is not trivial. Computer Vision and Natural Language Processing techniques have been advancing in extracting content from document images considering layout features. We propose a set of Layout Quadrant Tags (LayoutQT) as a new way of encoding layout information in textual embedding. We show that this enables a standard NLP pipeline to be significantly enhanced without requiring expensive mid or high-level multimodal fusion. Given that our focus is on developing a low computational cost solution, we focused our experiments on the AWD-LSTM neural network. We evaluated our method for page stream segmentation and document classification tasks on two datasets, Tobacco800 and RVL-CDIP. In the former, our method improved the F1 score from 97.9% to 99.1% and in the latter the F1 score went from 80.4% to 83.6%. Similar levels of performance improvement were also obtained when we applied LayoutQT with BERT.

Identifying learners' topical interests from social media content to enrich their course preferences in MOOCs using topic modeling and NLP techniques.

Interests play an essential role in the process of learning, thereby enriching learners 'interests will yield to an enhanced experience in MOOCs. Learners interact freely and spontaneously on social media through different forms of user-generated content which contain hidden information that reveals their real interests and preferences. In this paper, we aim to identify and extract the topical interest from the text content shared by learners on social media to enrich their course preferences in MOOCs. We apply NLP pipeline and topic modeling techniques to the textual feature using three well-known topic models: Latent Dirichlet Allocation, Latent Semantic Analysis, and BERTopic. The results of our experimentation have shown that BERTopic performed better on the scrapped dataset.

A NLP Pipeline for the Automatic Extraction of a Complete Microorganism's Picture from Microbiological Notes.

The Italian “Istituto Superiore di Sanità” (ISS) identifies hospital-acquired infections (HAIs) as the most frequent and serious complications in healthcare. HAIs constitute a real health emergency and, therefore, require decisive action from both local and national health organizations. Information about the causative microorganisms of HAIs is obtained from the results of microbiological cultures of specimens collected from infected body sites, but microorganisms’ names are sometimes reported only in the notes field of the culture reports. The objective of our work was to build a NLP-based pipeline for the automatic information extraction from the notes of microbiological culture reports. We analyzed a sample composed of 499 texts of notes extracted from 1 month of anonymized laboratory referral. First, our system filtered texts in order to remove nonmeaningful sentences. Thereafter, it correctly extracted all the microorganisms’ names according to the expert’s labels and linked them to a set of very important metadata such as the translations into national/international vocabularies and standard definitions. As the major result of our pipeline, the system extracts a complete picture of the microorganism.

Multicenter Validation of Natural Language Processing Algorithms for the Detection of Common Data Elements in Operative Notes for Total Hip Arthroplasty: Algorithm Development and Validation.

BackgroundNatural language processing (NLP) methods are powerful tools for extracting and analyzing critical information from free-text data. MedTaggerIE, an open-source NLP pipeline for information extraction based on text patterns, has been widely used in the annotation of clinical notes. A rule-based system, MedTagger-total hip arthroplasty (THA), developed based on MedTaggerIE, was previously shown to correctly identify the surgical approach, fixation, and bearing surface from the THA operative notes at Mayo Clinic.ObjectiveThis study aimed to assess the implementability, usability, and portability of MedTagger-THA at two external institutions, Michigan Medicine and the University of Iowa, and provide lessons learned for best practices.MethodsWe conducted iterative test-apply-refinement processes with three involved sites—the development site (Mayo Clinic) and two deployment sites (Michigan Medicine and the University of Iowa). Mayo Clinic was the primary NLP development site, with the THA registry as the gold standard. The activities at the two deployment sites included the extraction of the operative notes, gold standard development (Michigan: registry data; Iowa: manual chart review), the refinement of NLP algorithms on training data, and the evaluation of test data. Error analyses were conducted to understand language variations across sites. To further assess the model specificity for approach and fixation, we applied the refined MedTagger-THA to arthroscopic hip procedures and periacetabular osteotomy cases, as neither of these operative notes should contain any approach or fixation keywords.ResultsMedTagger-THA algorithms were implemented and refined independently for both sites. At Michigan, the study comprised THA-related notes for 2569 patient-date pairs. Before model refinement, MedTagger-THA algorithms demonstrated excellent accuracy for approach (96.6%, 95% CI 94.6%-97.9%) and fixation (95.7%, 95% CI 92.4%-97.6%). These results were comparable with internal accuracy at the development site (99.2% for approach and 90.7% for fixation). Model refinement improved accuracies slightly for both approach (99%, 95% CI 97.6%-99.6%) and fixation (98%, 95% CI 95.3%-99.3%). The specificity of approach identification was 88.9% for arthroscopy cases, and the specificity of fixation identification was 100% for both periacetabular osteotomy and arthroscopy cases. At the Iowa site, the study comprised an overall data set of 100 operative notes (50 training notes and 50 test notes). MedTagger-THA algorithms achieved moderate-high performance on the training data. After model refinement, the model achieved high performance for approach (100%, 95% CI 91.3%-100%), fixation (98%, 95% CI 88.3%-100%), and bearing surface (92%, 95% CI 80.5%-97.3%).ConclusionsHigh performance across centers was achieved for the MedTagger-THA algorithms, demonstrating that they were sufficiently implementable, usable, and portable to different deployment sites. This study provided important lessons learned during the model deployment and validation processes, and it can serve as a reference for transferring rule-based electronic health record models.

Evaluating Various Tokenizers for Arabic Text Classification

The first step in any NLP pipeline is to split the text into individual tokens. The most obvious and straightforward approach is to use words as tokens. However, given a large text corpus, representing all the words is not efficient in terms of vocabulary size. In the literature, many tokenization algorithms have emerged to tackle this problem by creating subwords, which in turn limits the vocabulary size in a given text corpus. Most tokenization techniques are language-agnostic, i.e., they do not incorporate the linguistic features of a given language. Not to mention the difficulty of evaluating such techniques in practice. In this paper, we introduce three new tokenization algorithms for Arabic and compare them to other three popular tokenizers using unsupervised evaluations. In addition, we compare all the six tokenizers by evaluating them on three supervised classification tasks: sentiment analysis, news classification and poem-meter classification, using six publicly available datasets. Our experiments show that none of the tokenization techniques is the best choice overall and that the performance of a given tokenization algorithm depends on many factors including the size of the dataset, nature of the task, and the morphology richness of the dataset. However, some tokenization techniques are better overall as compared to others on various text classification tasks.

Mapping SNOMED CT Codes to Semi-Structured Texts via an NLP Pipeline.

In the project presented here, we used NLP tools for annotating German medical trainings documents with SNOMED CT codes. Following research question was addressed: Is it possible to automate the annotation of training documents with an NLP pipeline especially designed for this task but requiring translation into English? The goal of our stakeholder, an institution responsible for the continuing education of physicians, was to facilitate the switch between different medical trainings programs by coding the same requirement with the same SNOMED CT code, even if the wording is different. We first describe how we chose the concrete NLP tools, after which the concrete steps for implementing our prototype are outlined: the NLP pipeline construction, the implementation, and the validation. We infer three important lessons from our results: (i) self-supervision is no free lunch and should be based on a sophisticated task, (ii) the translation via DeepL can be too context-dependent for a peculiar use case, and (iii) ontology extraction can increase efficiency as well as accuracy.

Connecting the dots in clinical document understanding with Relation Extraction at scale

<h2>Abstract</h2> We present a text mining framework based on top of the Spark NLP library - comprising of Named Entity Recognition (NER) and Relation Extraction (RE) models, which expands on previous work in three main ways. First, we release new RE model architectures that obtain state-of-the-art F1 scores on 5 out of 7 benchmark datasets. Second, we introduce a modular approach to train and stack multiple models in a single nlp pipeline in a production grade library with little coding. Third, we apply these models in practical applications including knowledge graph generation, prescription parsing, and robust ontology mapping.

Upon Improving the Performance of Localized Healthcare Virtual Assistants.

Virtual assistants are becoming popular in a variety of domains, responsible for automating repetitive tasks or allowing users to seamlessly access useful information. With the advances in Machine Learning and Natural Language Processing, there has been an increasing interest in applying such assistants in new areas and with new capabilities. In particular, their application in e-healthcare is becoming attractive and is driven by the need to access medically-related knowledge, as well as providing first-level assistance in an efficient manner. In such types of virtual assistants, localization is of utmost importance, since the general population (especially the aging population) is not familiar with the needed “healthcare vocabulary” to communicate facts properly; and state-of-practice proves relatively poor in performance when it comes to specialized virtual assistants for less frequently spoken languages. In this context, we present a Greek ML-based virtual assistant specifically designed to address some commonly occurring tasks in the healthcare domain, such as doctor’s appointments or distress (panic situations) management. We build on top of an existing open-source framework, discuss the necessary modifications needed to address the language-specific characteristics and evaluate various combinations of word embeddings and machine learning models to enhance the assistant’s behaviour. Results show that we are able to build an efficient Greek-speaking virtual assistant to support e-healthcare, while the NLP pipeline proposed can be applied in other (less frequently spoken) languages, without loss of generality.

Exploiting co-occurrence networks for classification of implicit inter-relationships in legal texts

The interpretation of any legal norm typically requires consideration of relationships between parts within the same piece of legislation. This work describes a general framework for the development of a system to identify and classify implicit inter-relationships between parts of a legal text. In particular, our approach demonstrates the usefulness of co-occurrence networks of terms, in a practical experimental setting based on an EU Regulation. First, a manual annotation task identify instances of different kinds of implicit links in the norm. In addition to a typical NLP pipeline, our framework includes a technique from Information Architecture, i.e. card sorting. Second, we construct co-occurrence networks of the law terms to derive graph metrics. Third, binary classification experiments identify the existence (and the type) of inter-relationships by using a Bag-of-Ngrams model integrated with network analysis features. The results demonstrate how the adoption of co-occurrence network features improves the identification of links, for all the classifiers here considered. This is encouraging toward a wider adoption of this kind of network analysis technique in legal informatics.

The Unified Medical Language System SPECIALIST Lexicon and Lexical Tools: Development and applications.

Natural language processing (NLP) plays a vital role in modern medical informatics. It converts narrative text or unstructured data into knowledge by analyzing and extracting concepts. A comprehensive lexical system is the foundation to the success of NLP applications and an essential component at the beginning of the NLP pipeline. The SPECIALIST Lexicon and Lexical Tools, distributed by the National Library of Medicine as one of the Unified Medical Language System Knowledge Sources, provides an underlying resource for many NLP applications. This article reports recent developments of 3 key components in the Lexicon. The core NLP operation of Unified Medical Language System concept mapping is used to illustrate the importance of these developments. Our objective is to provide generic, broad coverage and a robust lexical system for NLP applications. A novel multiword approach and other planned developments are proposed.

Development and validation of a pragmatic natural language processing approach to identifying falls in older adults in the emergency department

BackgroundFalls among older adults are both a common reason for presentation to the emergency department, and a major source of morbidity and mortality. It is critical to identify fall patients quickly and reliably during, and immediately after, emergency department encounters in order to deliver appropriate care and referrals. Unfortunately, falls are difficult to identify without manual chart review, a time intensive process infeasible for many applications including surveillance and quality reporting. Here we describe a pragmatic NLP approach to automating fall identification.MethodsIn this single center retrospective review, 500 emergency department provider notes from older adult patients (age 65 and older) were randomly selected for analysis. A simple, rules-based NLP algorithm for fall identification was developed and evaluated on a development set of 1084 notes, then compared with identification by consensus of trained abstractors blinded to NLP results.ResultsThe NLP pipeline demonstrated a recall (sensitivity) of 95.8%, specificity of 97.4%, precision of 92.0%, and F1 score of 0.939 for identifying fall events within emergency physician visit notes, as compared to gold standard manual abstraction by human coders.ConclusionsOur pragmatic NLP algorithm was able to identify falls in ED notes with excellent precision and recall, comparable to that of more labor-intensive manual abstraction. This finding offers promise not just for improving research methods, but as a potential for identifying patients for targeted interventions, quality measure development and epidemiologic surveillance.

A New Proposal for Evaluating Web Page Cleaning Tools

In this article, we tackle the problem of evaluation of Web Content Extraction tools. This task is seldom studied in the literature although it has important consequences on the linguistic processing of web-based corpora. Here, we compare two types of evaluation. Firstly, an intrinsic (content-based) evaluation which is carried out in a multilingual setting (five languages). Secondly, an extrinsic (task-based) evaluation on the same corpus by studying the effects of the cleaning step on the performances of an NLP pipeline. We show that in the intrinsic evaluation, the results are not consistent with extrinsic evaluation results. We also show that the results differ greatly in the studied languages. We conclude that the choice of a web page cleaning tool should be made with respect to the task that is tackled rather than the performances observed through the intrinsic evaluation scheme.

SEPIR: a semantic and personalised information retrieval tool for the public administration based on distributional semantics

This paper introduces a semantic and personalised information retrieval (SEPIR) tool for the public administration of Apulia Region. SEPIR, through semantic search and visualisation tools, enables the analysis of a large amount of unstructured data and the intelligent access to information. At the core of these functionalities is an NLP pipeline responsible for the WordSpace building and the key-phrase extraction. The WordSpace is the key component of the semantic search and personalisation algorithm. Moreover, key-phrases enrich the document representation of the retrieval system and are on the basis of the bubble charts, which provide a quick overview of the main concepts involved in a document collection. We show some of the key features of SEPIR in a use case where the personalisation technique re-ranks the set of relevant documents on the basis of the user's past queries and the visualisation tools provide the users with useful information about the analysed collection.

Testing Pre-Annotation to Help Non-Experts Identify Drug-Drug Interactions Mentioned in Drug Product Labeling

In this study, a system for allowing combination of textmining and crowdsourcing of annotation approaches for detection of DDIs from drug package inserts is presented. An annotation study was designed to evaluate expert versus non-expert curation performance, and the impact of NLP pre-annotation on precision and recall on both groups. The design and development of the system and annotation study, consisted of three stages. First, our existing NLP pipeline for DDI extraction was improved, and it was used to preannotate 208 drug product labels with drug mentions and DDIs. Secondly, a DDI machine readable representation scheme was created using the Annotation Ontolgy. This model allowed us to load the NLP preannotated drug label sections into our plugin for human curation created using the Annotation tool DOMEO. Finally, the annotation study was performed along with usability questionnaires for collecting qualitative feedback. To our knowledge, this is the first study in comparing experts and non-experts for pharmacokinetic DDI annotation. Results showed lower performance on non-experts compared with expert annotation without the use of NLP,and an improvement of non-expert annotation performance using the NER module of the NLP assistance. Simplification of the workflow for NLP assisted annotation is necessary for scaling ourapproach.

TagCurate: crowdsourcing the verification of biomedical annotations to mobile users

We present TagCurate, a distributed system that allows for disseminating biomedical annotations to users on Android-enabled devices for further verification. A web-based interface provides Task Managers with the ability to supervise the crowdsourcing process, as well as viewing the results gathered from the TagCurate Android app installed on the crowd's devices. We believe that the results of this research is beneficial to both curators and the NLP development communities. The efforts of expert curators will be efficiently allocated to resolving controversial annotations, while NLP pipeline developers can further train their algorithms from gold standard corpora solicited from a large group of contributors.