NLP Tools Research Articles

MT Embedded E-Learning in India - Challenges for NLP/AI

Indian languages are spoken by more than 90% of its population while most of the higher education happens in English medium. The policy makers in Indian government have realized that by introducing multilingual education electronically, they can reach out to the remotest corner of India and educate all in their mother tongue. The New Education Policy (NEP) draft just released by the government puts a heavy focus on mother tongues in education. The recent initiatives and focus on Natural Language Processing/ Artificial Intelligence (NLP/AI) in education through e-learning is not too surprising in this context. The paper presents the current initiatives in these directions by the government of India, surveys available NLP technologies particularly those for automatic translation of educational content developed by academia and industry and focuses on the Maithili language community. India's education needs are diverse and the success of e-learning depends heavily on the availability of necessary NLP tools in all languages. Almost all of major scheduled Indian languages are considered "resource-poor". While some of these languages may have the basic tools, they lack quality translation tools for delivering education in native language. The situation is more challenging in those languages where even the most basic resources and tools do not exist. Maithili - a language of Bihar and Nepal is such a language. The paper also presents an effort to develop MT resources and tools for Maithili and its application in delivering multilingual content for education.

Data from ‘The Dative Alternation Revisited: Fresh Insights from Contemporary British Spoken Data’

The dataset covers the so-called “dative alternation”. The dative alternation (also referred to as the ditransitive or double-object construction) refers to parallel constructions that have broadly similar meaning but different syntax: i. he gave it to the board” ii. “I gave her my old one” In i., the verb “give” takes a noun phrase (the pronoun “it”) and a prepositional phrase as arguments (“to the board”), while in ii. the verb takes two noun phrases (“her” and “my old one”) as arguments. In the dataset, we refer to i) as “VNPP” and ii) as “VNN”. We refer to the indirect object role as “recipient” (“her” in i. and “the board” in ii.) and the direct object as “theme” (“it” in i. and “my old one” in ii.). The dataset was collected from the Early-Access Subset (EAS) of the Spoken British National Corpus 2014 [1] for use in a sociolinguistic study of competing linguistic constructions [2]. The corpus is now publicly available via CQPweb at https://cqpweb.lancs.ac.uk/bnc2014spoken . The dative alternation is a topic of active research in linguistics, but few studies have made datasets available. Meta-studies, creation of specialised NLP tools, and comparisons of results will benefit from better access to this dataset.

NLP tools and processes of motivation

Abstract Neuro-linguistic programming (international acronym: NLP) is a special approach to understanding the communication process, triggering an attitude of curiosity, passion and dedication in studying the ways of improving and developing the behavior of human being, being not only an efficient system for stimulating spiritual evolution and increasing spiritual harmony, but also a great impetus for changing mentalities at the social level. also, it provides many useful means to deal with the real problems of today’s management and leadership. However, the information about NLP presented in the paper should not be interpreted as therapeutic indications, but this does not mean that it may be considered unnecessary.

Uma utilidade para o reconhecimento de topónimos em documentos medievais

Este artigo apresenta o método de construção duma ferramenta para a anotação de entidades geográficas mencionadas em textos medievais. A nova ferramenta foi desenvolvida a partir dos módulos de língua contemporânea do LinguaKit, pacote multilingue de ferramentas de PLN. Uma coleção de corpora anotados manualmente serviu de recurso para elaborar uma lista de topónimos medievais (gazetteers) e observar padrões para a melhora e implementação de novas regras de reconhecimento dos nomes de lugar. Depois da lista de entidades geográficas, os ativadores contextuais (triggers) foram o recurso determinante na melhora da abrangência. Para o produto final, fizeram-se também ajustes menores na procura de recolher os elementos mais comuns do léxico e os contextos gramaticais das entidades geográficas mencionadas. Ainda que muito trabalho fica por fazer na elaboração de listas para entidades não geográficas, na construção dum modelo de língua medieval e um lexicon específico, o novo módulo pode ser utilizado para anotar textos e mostra uma melhora significativa a respeito dos módulos previamente existentes.

Automatic Diacritics Restoration for Tunisian Dialect

Modern Standard Arabic, as well as Arabic dialect languages, are usually written without diacritics. The absence of these marks constitute a real problem in the automatic processing of these data by NLP tools. Indeed, writing Arabic without diacritics introduces several types of ambiguity. First, a word without diacratics could have many possible meanings depending on their diacritization. Second, undiacritized surface forms of an Arabic word might have as many as 200 readings depending on the complexity of its morphology [12]. In fact, the agglutination property of Arabic might produce a problem that can only be resolved using diacritics. Third, without diacritics a word could have many possible parts of speech (POS) instead of one. This is the case with the words that have the same spelling and POS tag but a different lexical sense, or words that have the same spelling but different POS tags and lexical senses [8]. Finally, there is ambiguity at the grammatical level (syntactic ambiguity). In this article, we propose the first work that investigates the automatic diacritization of Tunisian Dialect texts. We first describe our annotation guidelines and procedure. Then, we propose two major models, namely a statistical machine translation (SMT) and a discriminative model as a sequence classification task based on Conditional Random Fields (CRF). In the second approach, we integrate POS features to influence the generation of diacritics. Diacritics restoration was performed at both the word and the character levels. The results showed high scores of automatic diacritization based on the CRF system (Word Error Rate (WER) 21.44% for CRF and WER 34.6% for SMT).

A Comparative Study of NLP Topic Modeling Methods and Tools

A Comparative Study of NLP Topic Modeling Methods and Tools

Studying the history of the Arabic language: language technology and a large-scale historical corpus

Arabic is a widely-spoken language with a long and rich history, but existing corpora and language technology focus mostly on modern Arabic and its varieties. Therefore, studying the history of the language has so far been mostly limited to manual analyses on a small scale. In this work, we present a large-scale historical corpus of the written Arabic language, spanning 1400 years. We describe our efforts to clean and process this corpus using Arabic NLP tools, including the identification of reused text. We study the history of the Arabic language using a novel automatic periodization algorithm, as well as other techniques. Our findings confirm the established division of written Arabic into Modern Standard and Classical Arabic, and confirm other established periodizations, while suggesting that written Arabic may be divisible into still further periods of development.

Processing the Textual Information Using Open Natural Language Processing (NLP)

NLP is the application of computational technique and component of AI helps to analysis and synthesis of Standard English Language. It draws from many disciplines like computer science and computational linguistics to process and analyze large amount of natural language data. To identify the phrases in language that refers to specific types of entities and relations in text. The problem identified in N-gram approach is, balance weight is placed between in-frequent and frequent grams. It is efficient only for small amount of textual data. Thus, it is difficult to discover the named entities available in the corpus. The main aim of this paper is to overcome the limitations available in n-gram approach and facilitates to find desired pieces of entities by using Open NLP tool kit. It helps to store the information in a XML format that provides an easy way of querying and processing. Hence the information extracted from the web is an unstructured form. This approach is a promising way for processing the text in an efficient manner. To enhance the effectiveness of text mining, the researcher focused on the task of the NLP to discover knowledge from many unstructured text documents that leads to the largest available source of knowledge. The experiments and results shown in this paper with accuracy as 0.95 prove the confidence level is better than the N-grams approach.

Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review

Natural language processing (NLP) of symptoms from electronic health records (EHRs) could contribute to the advancement of symptom science. We aim to synthesize the literature on the use of NLP to process or analyze symptom information documented in EHR free-text narratives. Our search of 1964 records from PubMed and EMBASE was narrowed to 27 eligible articles. Data related to the purpose, free-text corpus, patients, symptoms, NLP methodology, evaluation metrics, and quality indicators were extracted for each study. Symptom-related information was presented as a primary outcome in 14 studies. EHR narratives represented various inpatient and outpatient clinical specialties, with general, cardiology, and mental health occurring most frequently. Studies encompassed a wide variety of symptoms, including shortness of breath, pain, nausea, dizziness, disturbed sleep, constipation, and depressed mood. NLP approaches included previously developed NLP tools, classification methods, and manually curated rule-based processing. Only one-third (n = 9) of studies reported patient demographic characteristics. NLP is used to extract information from EHR free-text narratives written by a variety of healthcare providers on an expansive range of symptoms across diverse clinical specialties. The current focus of this field is on the development of methods to extract symptom information and the use of symptom information for disease classification tasks rather than the examination of symptoms themselves. Future NLP studies should concentrate on the investigation of symptoms and symptom documentation in EHR free-text narratives. Efforts should be undertaken to examine patient characteristics and make symptom-related NLP algorithms or pipelines and vocabularies openly available.

Investigating Sentiment analysis using Clustering and NLP tools

Investigating Sentiment analysis using Clustering and NLP tools

A New Open Information Extraction System Using Sentence Difficulty Estimation

The World Wide Web has a considerable amount of information expressed using natural language. While unstructured text is often difficult for machines to understand, Open Information Extraction (OIE) is a relation-independent extraction paradigm designed to extract assertions directly from massive and heterogeneous corpora. Allocation of low-cost computational resources is a main demand for Open Relation Extraction (ORE) systems. A large number of ORE methods have been proposed recently, covering a wide range of NLP tools, from ``shallow'' (e.g., part-of-speech tagging) to ``deep'' (e.g., semantic role labeling). There is a trade-off between NLP tools depth versus efficiency (computational cost) of ORE systems. This paper describes a novel approach called Sentence Difficulty Estimator for Open Information Extraction (SDE-OIE) for automatic estimation of relation extraction difficulty by developing some difficulty classifiers. These classifiers dedicate the input sentence to an appropriate OIE extractor in order to decrease the overall computational cost. Our evaluations show that an intelligent selection of a proper depth of ORE systems has a significant improvement on the effectiveness and scalability of SDE-OIE. It avoids wasting resources and achieves almost the same performance as its constituent deep extractor in a more reasonable time.

MOSSA: a morpho-semantic knowledge extraction system for Arabic information retrieval

In this paper, we propose to exploit different morpho-semantic resources to enhance Arabic information retrieval (IR). We use standardised LMF Arabic dictionaries and Arabic corpora. Our goal by this communication is to take advantage of the different existing resources to extract useful knowledge for Arabic IR. We equally study the impact of the Arabic morphology on IR effectiveness. Several query expansion strategies are carried based on morphological, semantic and morpho-semantic relations. In addition, combining such knowledge is also studied and evaluated. We experiment the effect of short diacritics and part of speech (POS) disambiguation and tagging in the indexing step. A graph-based representation is used to formalise knowledge resources graph-based representation. This latter represents a powerful formalism to express semantics of texts and to support NLP tools and applications as IR. Several experimental comparisons are handled between the different used knowledge resources and the different carried IR approaches.

MOSSA: a morpho-semantic knowledge extraction system for Arabic information retrieval

In this paper, we propose to exploit different morpho-semantic resources to enhance Arabic information retrieval (IR). We use standardised LMF Arabic dictionaries and Arabic corpora. Our goal by this communication is to take advantage of the different existing resources to extract useful knowledge for Arabic IR. We equally study the impact of the Arabic morphology on IR effectiveness. Several query expansion strategies are carried based on morphological, semantic and morpho-semantic relations. In addition, combining such knowledge is also studied and evaluated. We experiment the effect of short diacritics and part of speech (POS) disambiguation and tagging in the indexing step. A graph-based representation is used to formalise knowledge resources graph-based representation. This latter represents a powerful formalism to express semantics of texts and to support NLP tools and applications as IR. Several experimental comparisons are handled between the different used knowledge resources and the different carried IR approaches.

Implementing a Portable Clinical NLP System with a Common Data Model - a Lisp Perspective.

This paper presents a Lisp architecture for a portable NLP system, termed LAPNLP, for processing clinical notes. LAPNLP integrates multiple standard, customized and in-house developed NLP tools. Our system facilitates portability across different institutions and data systems by incorporating an enriched Common Data Model (CDM) to standardize necessary data elements. It utilizes UMLS to perform domain adaptation when integrating generic domain NLP tools. It also features stand-off annotations that are specified by positional reference to the original document. We built an interval tree based search engine to efficiently query and retrieve the stand-off annotations by specifying positional requirements. We also developed a utility to convert an inline annotation format to stand-off annotations to enable the reuse of clinical text datasets with in-line annotations. We experimented with our system on several NLP facilitated tasks including computational phenotyping for lymphoma patients and semantic relation extraction for clinical notes. These experiments showcased the broader applicability and utility of LAPNLP.

SCAKE: Semantic Connectivity Aware Keyword Extraction

Keyword Extraction is an important task in several text analysis endeavours. In this paper, we present a critical discussion of the issues and challenges in graph-based keyword extraction methods, along with comprehensive empirical analysis. We propose a parameterless method for constructing graph of text that captures the contextual relation between words. A novel word scoring method is also proposed based on the connection between concepts. We demonstrate that both proposals are individually superior to those followed by the sate-of-the-art graph-based keyword extraction algorithms. Combination of the proposed graph construction and scoring methods leads to a novel, parameterless keyword extraction method (sCAKE) based on semantic connectivity of words in the document.Motivated by limited availability of NLP tools for several languages, we also design and present a language-agnostic keyword extraction (LAKE) method. We eliminate the need of NLP tools by using a statistical filter to identify candidate keywords before constructing the graph. We show that the resulting method is a competent solution for extracting keywords from documents of languages lacking sophisticated NLP support.

Exploring the Web and Semantic Knowledge-Driven Automatic Question Answering System

The growth of information retrieval from the web sources are increased day by day, proving an effective and efficient way to the user for retrieving relevant documents from the web is an art. Asking the right question and retrieving a right answer to the posted query is a service which provide by the Natural Language Processing. Question Answering System is one of the best ways to identify the candidate answer with high accuracy. The web and Semantic Knowledge Driven Question Answering System (QAS) used to determine the candidate answer for the posted query in the NLP tools. This method includes Query expansion techniques and entity linking method to identify the information source snippets with ontology structure, also ranking the sentences by applying conditional probability between query and Answer to identify the optimal answer from the web corpus. The result provides an exact answer with high accuracy than the baseline method.

From narrative descriptions to MedDRA: automagically encoding adverse drug reactions

ContextThe collection of narrative spontaneous reports is an irreplaceable source for the prompt detection of suspected adverse drug reactions (ADRs). In such task qualified domain experts manually revise a huge amount of narrative descriptions and then encode texts according to MedDRA standard terminology. The manual annotation of narrative documents with medical terminology is a subtle and expensive task, since the number of reports is growing up day-by-day. ObjectivesNatural Language Processing (NLP) applications can support the work of people responsible for pharmacovigilance. Our objective is to develop NLP algorithms and tools for the detection of ADR clinical terminology. Efficient applications can concretely improve the quality of the experts’ revisions. NLP software can quickly analyze narrative texts and offer an encoding (i.e., a list of MedDRA terms) that the expert has to revise and validate. MethodsMagiCoder, an NLP algorithm, is proposed for the automatic encoding of free-text descriptions into MedDRA terms. MagiCoder procedure is efficient in terms of computational complexity. We tested MagiCoder through several experiments. In the first one, we tested it on a large dataset of about 4500 manually revised reports, by performing an automated comparison between human and MagiCoder encoding. Moreover, we tested MagiCoder on a set of about 1800 reports, manually revised ex novo by some experts of the domain, who also compared automatic solutions with the gold reference standard. We also provide two initial experiments with reports written in English, giving a first evidence of the robustness of MagiCoder w.r.t. the change of the language. ResultsFor the current base version of MagiCoder, we measured an average recall and precision of 86.9% and 91.8%, respectively. ConclusionsFrom a practical point of view, MagiCoder reduces the time required for encoding ADR reports. Pharmacologists have only to review and validate the MedDRA terms proposed by the application, instead of choosing the right terms among the 70 K low level terms of MedDRA. Such improvement in the efficiency of pharmacologists’ work has a relevant impact also on the quality of the subsequent data analysis. We developed MagiCoder for the Italian pharmacovigilance language. However, our proposal is based on a general approach, not depending on the considered language nor the term dictionary.

Implementation on Framework for Restricted Domain Question Answering System using Advance NLP Tools & Software in

Implementation on Framework for Restricted Domain Question Answering System using Advance NLP Tools & Software in

A Medical Decision Support Tool Using Text-mining Techniques with Electronic Medical Records

Free-text clinical notes represent a vast amount of information which in the past has been un-analyzed data. In this paper we apply text-mining methods on the free-text in electronic medical records (EMRs) to define treatment options for patients with lower back pain. The goal of the project is to develop a generalized text-mining framework that can be used not only in the treatment of lower back pain, but any medical condition.
 The framework takes advantage of open-source algorithms for anonymization and the clinical NLP tool Apache Clinical Text Analysis and Knowledge Extraction System (cTAKES) to form structured data from clinical notes. The machine learning algorithm uses seven years of extracted clinical notes from the primary care physician to classify 20 patients’ pattern of back pain.
 With the small dataset provided, the algorithm managed to achieve diagnosis accuracy of up to 100%. The twenty-patient dataset was simply too homogenous and small to make statistical claims for sensitivity and specificity. However, the system shows indicators of satisfactory performance, and we are trying to extract more data of patients who do not have back pain to be able to validate our system better.

Discovering and learning sensational episodes of news events

In this paper, we study the problem of discovering and learning sensational episodes of news events. A sensational episode of news events is in the form of lhs → rhs, where lhs is an antecedent event, rhs is a consequent event, and rhs often happens shortly after lhs. Such pairs of co-occurring news events within short period, while not necessarily bearing causal relationship between each other, are possible essential to media since they deliberately seek and broadcast examples of uncommon events to fascinate crowd attentions. First, to find all frequent episodes, we propose an efficient algorithm, MEELO, which significantly outperforms conventional algorithms. There can be a large number of frequent episodes. We rank them by their sensational effect from the perspectives of news audience, through learning from manually labeled examples. Instead of limiting ourselves to any individual subjective measure of sensational effect, we utilize a learning-to-rank approach that exploits multiple features to capture the sensational effect of a news episode from various aspects. NLP tools combined with knowledge bases are used in extracting and aggregating news events from news text. Experiments on real data verify our approach’s efficiency and effectiveness.