Information Extraction System Research Articles

An annotated corpus of clinical trial publications supporting schema-based relational information extraction

BackgroundThe evidence-based medicine paradigm requires the ability to aggregate and compare outcomes of interventions across different trials. This can be facilitated and partially automatized by information extraction systems. In order to support the development of systems that can extract information from published clinical trials at a fine-grained and comprehensive level to populate a knowledge base, we present a richly annotated corpus at two levels. At the first level, entities that describe components of the PICO elements (e.g., population’s age and pre-conditions, dosage of a treatment, etc.) are annotated. The second level comprises schema-level (i.e., slot-filling templates) annotations corresponding to complex PICO elements and other concepts related to a clinical trial (e.g. the relation between an intervention and an arm, the relation between an outcome and an intervention, etc.).ResultsThe final corpus includes 211 annotated clinical trial abstracts with substantial agreement between annotators at the entity and scheme level. The mean Kappa value for the glaucoma and T2DM corpora was 0.74 and 0.68, respectively, for single entities. The micro-averaged F1 score to measure inter-annotator agreement for complex entities (i.e. slot-filling templates) was 0.81.The BERT-base baseline method for entity recognition achieved average micro- F1 scores of 0.76 for glaucoma and 0.77 for diabetes with exact matching.ConclusionsIn this work, we have created a corpus that goes beyond the existing clinical trial corpora, since it is annotated in a schematic way that represents the classes and properties defined in an ontology. Although the corpus is small, it has fine-grained annotations and could be used to fine-tune pre-trained machine learning models and transformers to the specific task of extracting information about clinical trial abstracts.For future work, we will use the corpus for training information extraction systems that extract single entities, and predict template slot-fillers (i.e., class data/object properties) to populate a knowledge base that relies on the C-TrO ontology for the description of clinical trials. The resulting corpus and the code to measure inter-annotation agreement and the baseline method are publicly available at https://zenodo.org/record/6365890.

Automated medical chart review for breast cancer outcomes research: a novel natural language processing extraction system

BackgroundManually extracted data points from health records are collated on an institutional, provincial, and national level to facilitate clinical research. However, the labour-intensive clinical chart review process puts an increasing burden on healthcare system budgets. Therefore, an automated information extraction system is needed to ensure the timeliness and scalability of research data.MethodsWe used a dataset of 100 synoptic operative and 100 pathology reports, evenly split into 50 reports in training and test sets for each report type. The training set guided our development of a Natural Language Processing (NLP) extraction pipeline system, which accepts scanned images of operative and pathology reports. The system uses a combination of rule-based and transfer learning methods to extract numeric encodings from text. We also developed visualization tools to compare the manual and automated extractions. The code for this paper was made available on GitHub.ResultsA test set of 50 operative and 50 pathology reports were used to evaluate the extraction accuracies of the NLP pipeline. Gold standard, defined as manual extraction by expert reviewers, yielded accuracies of 90.5% for operative reports and 96.0% for pathology reports, while the NLP system achieved overall 91.9% (operative) and 95.4% (pathology) accuracy. The pipeline successfully extracted outcomes data pertinent to breast cancer tumor characteristics (e.g. presence of invasive carcinoma, size, histologic type), prognostic factors (e.g. number of lymph nodes with micro-metastases and macro-metastases, pathologic stage), and treatment-related variables (e.g. margins, neo-adjuvant treatment, surgical indication) with high accuracy. Out of the 48 variables across operative and pathology codebooks, NLP yielded 43 variables with F-scores of at least 0.90; in comparison, a trained human annotator yielded 44 variables with F-scores of at least 0.90.ConclusionsThe NLP system achieves near-human-level accuracy in both operative and pathology reports using a minimal curated dataset. This system uniquely provides a robust solution for transparent, adaptable, and scalable automation of data extraction from patient health records. It may serve to advance breast cancer clinical research by facilitating collection of vast amounts of valuable health data at a population level.

NLP-Based Query-Answering System for Information Extraction from Building Information Models

The construction industry is information-intensive, and building information modeling (BIM) has been proposed as an information source for supporting decision making by construction project team members in the architecture, engineering, construction, and operation (AECO) industry. Because building information models contain more building data, further use of the aggregated building information to support construction and operation activities has become important. In Industry 4.0, similar-to-real-life virtual assistants, e.g., Apple’s Siri and Google Assistant, are becoming ever more popular. This research developed a query-answering (QA) system for BIM information extraction (IE) by using natural language processing (NLP) methods to build a virtual assistant for construction project team members. The architecture of the developed QA system for BIM IE consists of three major modules: natural language understanding, IE, and natural language generation. A Python-based prototype application was developed based on the architecture of the QA system for BIM IE to evaluate functionalities of the developed QA system using several BIM/industry foundation classes (IFC) models. Seven building information models and 127 test queries were utilized to evaluate the accuracy of the developed QA system for BIM IE. The experimental results indicated that the developed QA system for BIM IE achieved an 81.9 accuracy score. The developed NLP-based QA system for BIM is valid to provide relatively accurate answers based on natural language queries. The contributions of this research facilitate the development of virtual assistants in the AECO industry, and the architecture of the developed QA system can be extended to queries in other areas.

Natural Language Processing-Assisted Literature Retrieval and Analysis for Combination Therapy in Cancer.

Despite advances in molecular therapeutics, few anticancer agents achieve durable responses. Rational combinations using two or more anticancer drugs have the potential to achieve a synergistic effect and overcome drug resistance, enhancing antitumor efficacy. A publicly accessible biomedical literature search engine dedicated to this domain will facilitate knowledge discovery and reduce manual search and review. We developed RetriLite, an information retrieval and extraction framework that leverages natural language processing and domain-specific knowledgebase to computationally identify highly relevant papers and extract key information. The modular architecture enables RetriLite to benefit from synergizing information retrieval and natural language processing techniques while remaining flexible to customization. We customized the application and created an informatics pipeline that strategically identifies papers that describe efficacy of using combination therapies in clinical or preclinical studies. In a small pilot study, RetriLite achieved an F1 score of 0.93. A more extensive validation experiment was conducted to determine agents that have enhanced antitumor efficacy in vitro or in vivo with poly (ADP-ribose) polymerase inhibitors: 95.9% of the papers determined to be relevant by our application were true positive and the application's feature of distinguishing a clinical paper from a preclinical paper achieved an accuracy of 97.6%. Interobserver assessment was conducted, which resulted in a 100% concordance. The data derived from the informatics pipeline have also been made accessible to the public via a dedicated online search engine with an intuitive user interface. RetriLite is a framework that can be applied to establish domain-specific information retrieval and extraction systems. The extensive and high-quality metadata tags along with keyword highlighting facilitate information seekers to more effectively and efficiently discover knowledge in the combination therapy domain.

A spatio-temporal emotional framework for knowledge extraction and mining in digital humanities

PurposeThis paper aims to construct a spatio-temporal emotional framework (STEF) for digital humanities from a quantitative perspective, applying knowledge extraction and mining technology to promote innovation of humanities research paradigm and method.Design/methodology/approachThe proposed STEF uses methods of information extraction, sentiment analysis and geographic information system to achieve knowledge extraction and mining. STEF integrates time, space and emotional elements to visualize the spatial and temporal evolution of emotions, which thus enriches the analytical paradigm in digital humanities.FindingsThe case study shows that STEF can effectively extract knowledge from unstructured texts in the field of Chinese Qing Dynasty novels. First, STEF introduces the knowledge extraction tools – MARKUS and DocuSky – to profile character entities and perform plots extraction. Second, STEF extracts the characters' emotional evolutionary trajectory from the temporal and spatial perspective. Finally, the study draws a spatio-temporal emotional path figure of the leading characters and integrates the corresponding plots to analyze the causes of emotion fluctuations.Originality/valueThe STEF is constructed based on the “spatio-temporal narrative theory” and “emotional narrative theory”. It is the first framework to integrate elements of time, space and emotion to analyze the emotional evolution trajectories of characters in novels. The execuability and operability of the framework is also verified with a case novel to suggest a new path for quantitative analysis of other novels.

Design and implementation of information extraction system for scientific literature using fine-tuned deep learning models

This paper presents an overview of a quality scoring system that utilizes pre-trained deep neural network models. Two types of DL models, a classification and extractive question answering (EQA) models are used to implement components of the system. The abstracts of the scientific literature are classified into two groups, in-vivo and in-vitro, and a question and answering model architecture is constructed for extracting the following types of information (animal type, the number of animals, exposure dose, and signal frequency). The Bidirectional Encoder Representations of Transformers (BERT) model pre-trained with a large text corpus is used as our baseline model for classification and EQA tasks. The models are fine-tuned with 455 EMF-related research papers. In our experiments, the fine-tuned model showed improved performance on EQA tasks for the four-categories of questions compared to the baseline, and it also showed improvements on similar questions that were not used in training. This suggests the importance of retraining of deep learning model specifically in some areas requiring domain expertise such as scientific papers. However, additional research is needed on some implementation issues, in such cases where there are still multiple answers, or where there is no answer given in a context.

Hand Pronation-Supination Movement as a Proxy for Remotely Monitoring Gait and Posture Stability in Parkinson's Disease.

The Unified Parkinson’s Disease Rating Scale (UPDRS) is a subjective Parkinson’s Disease (PD) physician scoring/monitoring system. To date, there is no single upper limb wearable/non-contact system that can be used objectively to assess all UPDRS-III motor system subgroups (i.e., tremor (T), rigidity (R), bradykinesia (B), gait and posture (GP), and bulbar anomalies (BA)). We evaluated the use of a non-contact hand motion tracking system for potential extraction of GP information using forearm pronation–supination (P/S) motion parameters (speed, acceleration, and frequency). Twenty-four patients with idiopathic PD participated, and their UPDRS data were recorded bilaterally by physicians. Pearson’s correlation, regression analyses, and Monte Carlo validation was conducted for all combinations of UPDRS subgroups versus motion parameters. In the 262,125 regression models that were trained and tested, the models within 1% of the lowest error showed that the frequency of P/S contributes to approximately one third of all models; while speed and acceleration also contribute significantly to the prediction of GP from the left-hand motion of right handed patients. In short, the P/S better indicated GP when performed with the non-dominant hand. There was also a significant negative correlation (with medium to large effect size, range: 0.3–0.58) between the P/S speed and the single BA score for both forearms and combined UPDRS score for the dominant hand. This study highlights the potential use of wearable or non-contact systems for forearm P/S to remotely monitor and predict the GP information in PD.

Why does the president tweet this? Discovering reasons and contexts for politicians’ tweets from news articles

Politicians’ tweets can have important political and economic implications. However, limited context makes it hard for readers to instantly and precisely understand them, especially from a causal perspective. The triggers for these tweets may have been reported in news prior to the tweets, but simply finding similar news articles would not serve the purpose, given the following reasons. First, readers may only be interested in finding the reasons and contexts (we call causal backgrounds) for a certain part of a tweet. Intuitively, such content would be politically relevant and accord with public’s recent attention, which is not usually reflected within the context. Besides, the content should be human-readable, while the noisy and informal nature of tweets hinders regular Open Information Extraction systems. Second, similarity does not capture causality and the causality between tweet contents and news contents is beyond the scopes of causality extraction tools. Meanwhile, it will be non-trivial to construct a high-quality tweet-to-intent dataset.We propose the first end-to-end framework for discovering causal backgrounds of politicians’ tweets by: 1. Designing an Open IE system considering rule-free representations for tweets; 2. Introducing sources like Wikipedia linkage and edit history to identify focal contents; 3. Finding implicit causalities between different contexts using explicit causalities learned elsewhere. We curate a comprehensive dataset of interpretations from political journalists for 533 tweets from 5 US politicians. On average, we obtain the correct answers within top-2 recommendations. We make our dataset and framework code publicly available.

Arabic open information extraction system using dependency parsing

<p>Arabic is a Semitic language and one of the most natural languages distinguished by the richness in morphological enunciation and derivation. This special and complex nature makes extracting information from the Arabic language difficult and always needs improvement. Open information extraction systems (OIE) have been emerged and used in different languages, especially in English. However, it has almost not been used for the Arabic language. Accordingly, this paper aims to introduce an OIE system that extracts the relation tuple from Arabic web text, exploiting Arabic dependency parsing and thinking carefully about all possible text relations. Based on clause types' propositions as extractable relations and constituents' grammatical functions, the identities of corresponding clause types are established. The proposed system named Arabic open information extraction(AOIE) can extract highly scalable Arabic text relations while being domain independent. Implementing the proposed system handles the problem using supervised strategies while the system relies on unsupervised extraction strategies. Also, the system has been implemented in several domains to avoid information extraction in a specific field. The results prove that the system achieves high efficiency in extracting clauses from large amounts of text.</p>

A Unified Framework of Medical Information Annotation and Extraction for Chinese Clinical Text

Medical information extraction consists of a group of natural language processing (NLP) tasks, which collaboratively convert clinical text to pre-defined structured formats. Current state-of-the-art (SOTA) NLP models are highly integrated with deep learning techniques and thus require massive annotated linguistic data. This study presents an engineering framework of medical entity recognition, relation extraction and attribute extraction, which are unified in annotation, modeling and evaluation. Specifically, the annotation scheme is comprehensive, and compatible between tasks, especially for the medical relations. The resulted annotated corpus includes 1,200 full medical records (or 18,039 broken-down documents), and achieves inter-annotator agreements (IAAs) of 94.53%, 73.73% and 91.98% F 1 scores for the three tasks. Three task-specific neural network models are developed within a shared structure, and enhanced by SOTA NLP techniques, i.e., pre-trained language models. Experimental results show that the system can retrieve medical entities, relations and attributes with F 1 scores of 93.47%, 67.14% and 90.89%, respectively. This study, in addition to our publicly released annotation scheme and code, provides solid and practical engineering experience of developing an integrated medical information extraction system.

Real-World Actionable Information Extraction System (Aies) for Social Media

Real-World Actionable Information Extraction System (Aies) for Social Media

From Tokenization to Self-Supervision: Building a High-Performance Information Extraction System for Chemical Reactions in Patents.

Chemical reactions and experimental conditions are fundamental information for chemical research and pharmaceutical applications. However, the latest information of chemical reactions is usually embedded in the free text of patents. The rapidly accumulating chemical patents urge automatic tools based on natural language processing (NLP) techniques for efficient and accurate information extraction. This work describes the participation of the Melax Tech team in the CLEF 2020—ChEMU Task of Chemical Reaction Extraction from Patent. The task consisted of two subtasks: (1) named entity recognition to identify compounds and different semantic roles in the chemical reaction and (2) event extraction to identify event triggers of chemical reaction and their relations with the semantic roles recognized in subtask 1. To build an end-to-end system with high performance, multiple strategies tailored to chemical patents were applied and evaluated, ranging from optimizing the tokenization, pre-training patent language models based on self-supervision, to domain knowledge-based rules. Our hybrid approaches combining different strategies achieved state-of-the-art results in both subtasks, with the top-ranked F1 of 0.957 for entity recognition and the top-ranked F1 of 0.9536 for event extraction, indicating that the proposed approaches are promising.

Information extraction from scanned invoice images using text analysis and layout features

While storing invoice content as metadata to avoid paper document processing may be the future trend, almost all of daily issued invoices are still printed on paper or generated in digital formats such as PDFs. In this paper, we introduce the OCRMiner system for information extraction from scanned document images which is based on text analysis techniques in combination with layout features to extract indexing metadata of (semi-)structured documents. The system is designed to process the document in a similar way a human reader uses, i.e. to employ different layout and text attributes in a coordinated decision. The system consists of a set of interconnected modules that start with (possibly erroneous) character-based output from a standard OCR system and allow to apply different techniques and to expand the extracted knowledge at each step. Using an open source OCR, the system is able to recover the invoice data in 90% for English and in 88% for the Czech set.

A precision‐preferred comprehensive information extraction system for clinical articles in traditional Chinese Medicine

This study established a precision-preferred system specially designed for the data extraction of traditional Chinese medicine (TCM) articles, providing foundational data for subsequent clinical article analysis and synthesis of TCM clinical evidence. Information extraction is commonly used in many fields to identify relevant concepts and the relationship between pairs of concepts from the vast information sources. Previous studies that performed information extraction primarily focused on scattering targeted fields to achieve a balance between precision and recall. Therefore, this study aims to create a comprehensive information extraction system for TCM articles. This system will extract all relevant information from research articles on a broad research field, including the 11 diseases that can be efficiently treated with TCM, with high precision and efficient measurement to address bias in every study. It covers the most essential information related to patients, interventions, comparisons, outcomes, and study design (PICOS) principles in TCM clinical trials. This system covers 34 target fields on 14 topics. Impediments such as the various typesetting of TCM clinical articles were managed by a hybrid of machine vision and optical character recognition. Thus, TCM researchers can be spared of laborious, unscalable, and inefficient manual extraction processes. Our system could also enhance TCM researcher awareness of frequently missing information or TCM clinical trial design methods that could introduce bias, by analyzing the overall information integrity of TCM clinical articles, which is beneficial for future research designs.

Profile generation from web sources: an information extraction system

Profile generation from web sources: an information extraction system

A Policy-Driven Approach to Secure Extraction of COVID-19 Data From Research Papers.

The entire scientific and academic community has been mobilized to gain a better understanding of the COVID-19 disease and its impact on humanity. Most research related to COVID-19 needs to analyze large amounts of data in very little time. This urgency has made Big Data Analysis, and related questions around the privacy and security of the data, an extremely important part of research in the COVID-19 era. The White House OSTP has, for example, released a large dataset of papers related to COVID research from which the research community can extract knowledge and information. We show an example system with a machine learning-based knowledge extractor which draws out key medical information from COVID-19 related academic research papers. We represent this knowledge in a Knowledge Graph that uses the Unified Medical Language System (UMLS). However, publicly available studies rely on dataset that might have sensitive data. Extracting information from academic papers can potentially leak sensitive data, and protecting the security and privacy of this data is equally important. In this paper, we address the key challenges around the privacy and security of such information extraction and analysis systems. Policy regulations like HIPAA have updated the guidelines to access data, specifically, data related to COVID-19, securely. In the US, healthcare providers must also comply with the Office of Civil Rights (OCR) rules to protect data integrity in matters like plasma donation, media access to health care data, telehealth communications, etc. Privacy policies are typically short and unstructured HTML or PDF documents. We have created a framework to extract relevant knowledge from the health centers’ policy documents and also represent these as a knowledge graph. Our framework helps to understand the extent to which individual provider policies comply with regulations and define access control policies that enforce the regulation rules on data in the knowledge graph extracted from COVID-related papers. Along with being compliant, privacy policies must also be transparent and easily understood by the clients. We analyze the relative readability of healthcare privacy policies and discuss the impact. In this paper, we develop a framework for access control decisions that uses policy compliance information to securely retrieve COVID data. We show how policy compliance information can be used to restrict access to COVID-19 data and information extracted from research papers.

Construction of Genealogical Knowledge Graphs From Obituaries: Multitask Neural Network Extraction System.

BackgroundGenealogical information, such as that found in family trees, is imperative for biomedical research such as disease heritability and risk prediction. Researchers have used policyholder and their dependent information in medical claims data and emergency contacts in electronic health records (EHRs) to infer family relationships at a large scale. We have previously demonstrated that online obituaries can be a novel data source for building more complete and accurate family trees.ObjectiveAiming at supplementing EHR data with family relationships for biomedical research, we built an end-to-end information extraction system using a multitask-based artificial neural network model to construct genealogical knowledge graphs (GKGs) from online obituaries. GKGs are enriched family trees with detailed information including age, gender, death and birth dates, and residence.MethodsBuilt on a predefined family relationship map consisting of 4 types of entities (eg, people’s name, residence, birth date, and death date) and 71 types of relationships, we curated a corpus containing 1700 online obituaries from the metropolitan area of Minneapolis and St Paul in Minnesota. We also adopted data augmentation technology to generate additional synthetic data to alleviate the issue of data scarcity for rare family relationships. A multitask-based artificial neural network model was then built to simultaneously detect names, extract relationships between them, and assign attributes (eg, birth dates and death dates, residence, age, and gender) to each individual. In the end, we assemble related GKGs into larger ones by identifying people appearing in multiple obituaries.ResultsOur system achieved satisfying precision (94.79%), recall (91.45%), and F-1 measures (93.09%) on 10-fold cross-validation. We also constructed 12,407 GKGs, with the largest one made up of 4 generations and 30 people.ConclusionsIn this work, we discussed the meaning of GKGs for biomedical research, presented a new version of a corpus with a predefined family relationship map and augmented training data, and proposed a multitask deep neural system to construct and assemble GKGs. The results show our system can extract and demonstrate the potential of enriching EHR data for more genetic research. We share the source codes and system with the entire scientific community on GitHub without the corpus for privacy protection.

Extracting Meta Statements from the Blogosphere

Information extraction systems have been recently proposed for organizing and exploring content in large online text corpora as information networks. In such networks, the nodes are named entities (e.g., people, organizations) while the edges correspond to statements indicating relations among such entities. To date, such systems extract rather primitive networks, capturing only those relations which are expressed by direct statements. In many applications, it is useful to also extract more subtle relations which are often expressed as meta statements in the text. These can, for instance provide the context for a statement (e.g., “Google acquired YouTube on October 2006”), or repercussion about a statement (e.g., “The US condemned Russia’s invasion of Georgia”). In this work, we report on a system for extracting relations expressed in both direct statements as well as in meta statements. We propose a method based on Conditional Random Fields that explores syntactic features to extract both kinds of statements seamlessly. We follow the Open Information Extraction paradigm, where a classifier is trained to recognize any type of relation instead of specific ones. Finally, our results show substantial improvements over a state-of-the-art information extraction system, both in terms of accuracy and, especially, recall.

MT-clinical BERT: scaling clinical information extraction with multitask learning.

Clinical notes contain an abundance of important, but not-readily accessible, information about patients. Systems that automatically extract this information rely on large amounts of training data of which there exists limited resources to create. Furthermore, they are developed disjointly, meaning that no information can be shared among task-specific systems. This bottleneck unnecessarily complicates practical application, reduces the performance capabilities of each individual solution, and associates the engineering debt of managing multiple information extraction systems. We address these challenges by developing Multitask-Clinical BERT: a single deep learning model that simultaneously performs 8 clinical tasks spanning entity extraction, personal health information identification, language entailment, and similarity by sharing representations among tasks. We compare the performance of our multitasking information extraction system to state-of-the-art BERT sequential fine-tuning baselines. We observe a slight but consistent performance degradation in MT-Clinical BERT relative to sequential fine-tuning. These results intuitively suggest that learning a general clinical text representation capable of supporting multiple tasks has the downside of losing the ability to exploit dataset or clinical note-specific properties when compared to a single, task-specific model. We find our single system performs competitively with all state-the-art task-specific systems while also benefiting from massive computational benefits at inference.

Natural language processing for automated annotation of medication mentions in primary care visit conversations.

ObjectivesThe objective of this study is to build and evaluate a natural language processing approach to identify medication mentions in primary care visit conversations between patients and physicians.Materials and MethodsEight clinicians contributed to a data set of 85 clinic visit transcripts, and 10 transcripts were randomly selected from this data set as a development set. Our approach utilizes Apache cTAKES and Unified Medical Language System controlled vocabulary to generate a list of medication candidates in the transcribed text and then performs multiple customized filters to exclude common false positives from this list while including some additional common mentions of the supplements and immunizations.ResultsSixty-five transcripts with 1121 medication mentions were randomly selected as an evaluation set. Our proposed method achieved an F-score of 85.0% for identifying the medication mentions in the test set, significantly outperforming existing medication information extraction systems for medical records with F-scores ranging from 42.9% to 68.9% on the same test set.DiscussionOur medication information extraction approach for primary care visit conversations showed promising results, extracting about 27% more medication mentions from our evaluation set while eliminating many false positives in comparison to existing baseline systems. We made our approach publicly available on the web as an open-source software.ConclusionIntegration of our annotation system with clinical recording applications has the potential to improve patients’ understanding and recall of key information from their clinic visits, and, in turn, to positively impact health outcomes.