Natural Language Processing Toolkit Research Articles

The ENACT Network is Acting on Housing Instability and the Unhoused Using the Open Health Natural Language Processing Toolkit

The ENACT Network is Acting on Housing Instability and the Unhoused Using the Open Health Natural Language Processing Toolkit

Use of Natural Language Processing to Identify Sexual and Reproductive Health Information in Clinical Text.

This study aimed to enable clinical researchers without expertise in natural language processing (NLP) to extract and analyze information about sexual and reproductive health (SRH), or other sensitive health topics, from large sets of clinical notes. (1) We retrieved text from the electronic health record as individual notes. (2) We segmented notes into sentences using one of scispaCy's NLP toolkits. (3) We exported sentences to the labeling application Watchful and annotated subsets of these as relevant or irrelevant to various SRH categories by applying a combination of regular expressions and manual annotation. (4) The labeled sentences served as training data to create machine learning models for classifying text; specifically, we used spaCy's default text classification ensemble, comprising a bag-of-words model and a neural network with attention. (5) We applied each model to unlabeled sentences to identify additional references to SRH with novel relevant vocabulary. We used this information and repeated steps 3 to 5 iteratively until the models identified no new relevant sentences for each topic. Finally, we aggregated the labeled data for analysis. This methodology was applied to 3,663 Child Neurology notes for 971 female patients. Our search focused on six SRH categories. We validated the approach using two subject matter experts, who independently labeled a sample of 400 sentences. Cohen's kappa values were calculated for each category between the reviewers (menstruation: 1, sexual activity: 0.9499, contraception: 0.9887, folic acid: 1, teratogens: 0.8864, pregnancy: 0.9499). After removing the sentences on which reviewers did not agree, we compared the reviewers' labels to those produced via our methodology, again using Cohen's kappa (menstruation: 1, sexual activity: 1, contraception: 0.9885, folic acid: 1, teratogens: 0.9841, pregnancy: 0.9871). Our methodology is reproducible, enables analysis of large amounts of text, and has produced results that are highly comparable to subject matter expert manual review.

Using natural language processing to generate a large-scale database of aortic stenosis with long-term follow-up: the CASPER (cogstack aortic stenosis patient electronic registry) database

Abstract Background Aortic stenosis (AS) is one of the most common heart valve diseases in the Western World, with a prevalence of approximately 4% in patients over 70 years old. High quality observational data can provide insight into characteristics that define patient trajectories and inform the design of appropriately powered randomised trials. Objective The aim of this study is to develop a data pipeline to generate a large database containing information of an AS cohort in a hospital by analysing both structured and unstructured (free text) data in routinely collected electronic heath records (EHR). The unstructured data are processed using natural language processing (NLP, an artificial intelligence technique that allows machines to recognize clinical concepts in EHR with contextualisation and linked to SNOMED ontology). Methods The open-source CogStack retrieval system is used to collect clinical notes of AS patients. The natural language processing toolkit, MedCAT, is used to identify clinical concepts in the form of SNOMED terms in clinical notes. In addition to patient demographic data (including ethnicity and markers of social deprivation), a range of clinical characteristics, co-morbidities, medications and procedural interventions are extracted. Validation of the dataset was performed using 3 existing, independent data sources (structured echocardiogram reports, surgical aortic valve replacement (AVR) operation notes, transcatheter intervention (TAVI) reports). Results We identified 7,451 patients with AS from our EHR, of whom 5,754 had an echocardiogram at our centre. The mean duration of follow up was 5 years. Of those with an echocardiogram at our centre, we were able to extract data on AS severity in 91% and left ventricular function in 89%. Manual validation of procedural data suggests a sensitivity of 98% to detect TAVI and 97% to detect AVR procedures. Kaplan-Meier curves for automated detection of patient mortality are consistent with the known survival data for patients with AS. Conclusion We developed and validated an automated NLP-enabled data pipeline to identify and characterise a single retrospective centre cohort of AS patients from routinely collected EHR across the spectrum of severity, with long term mortality data. This resource has the potential to provide insights toward several outstanding questions relating to progression of AS, and the influence of ethnicity and social deprivation without the need for conventional dedicated databases. The EHR-based database also has the potential to guide the design of future randomised control trials.Data Pipeline

Using a Natural Language Processing Toolkit to Classify Patient Charts by Psychiatric Diagnosis

Using a Natural Language Processing Toolkit to Classify Patient Charts by Psychiatric Diagnosis

An open-source natural language processing toolkit to support software development: addressing automatic bug detection, code summarisation and code search.

This paper aims to introduce the innovative work carried out in the Horizon 2020 DECODER project - acronym for "DEveloper COmpanion for Documented and annotatEd code Reference" - (Grant Agreement no. 824231) by linking the fields of natural language processing (NLP) and software engineering. The project as a whole addresses the development of a framework, namely the Persistent Knowledge Monitor (PKM), that acts as a central infrastructure to store, access, and trace all the data, information and knowledge related to a given software or ecosystem. This meta-model defines the knowledge base that can be queried and analysed by all the tools integrated and developed in DECODER. Besides, the DECODER project offers a friendly user interface where each of the predefined three roles (i.e., developers, maintainers and reviewers) can access and query the PKM with their personal accounts. The paper focuses on the NLP tools developed and integrated in the PKM, namely the deep learning models developed to perform variable misuse, code summarisation and semantic parsing. These were developed under a common work package - "Activities for the developer" - intended to precisely target developers, who can perform tasks such as detection of bugs, automatic generation of documentation for source code and generation of code snippets from natural languages instructions, among the multiple functionalities that DECODER offers. These tools assist and help the developers in the daily work, by increasing their productivity and avoiding loss of time in tedious tasks such as manual bug detection. Training and validation were conducted for four use cases in Java, C and C++ programming languages in order to evaluate the performance, suitability, usability, etc. of the developed tools.

Did the emergence of the Omicron variant impact the COVID-19 vaccine acceptance in the Arab world? A Twitter-based sentiment analysis

Background: Low vaccine acceptance constitutes a major obstacle for achieving herd immunity against coronavirus disease 2019 (COVID-19). Data on COVID-19 show a significant rise in the number of cumulative cases in Arab countries where vaccine uptake is lower than the uptake at the global level for the 2021-2022 period. The aim of this study is to analyze sentiments related to vaccine acceptance before and after the emergence of the COVID-19 Omicron variant in the Arab-speaking world. Methods: Tweets were collected before and after the Omicron emergence period in the Arab-speaking world using Python-enabled CAMeL Tools (Arabic Natural Language Processing toolkit) libraries. A sentiment analysis on COVID-19 vaccines using Arabic hashtags and a sentiment analysis by city were conducted. Results: Two data sets pre-Omicron and post-Omicron tweets exhibited a difference in sentiments marking a shift towards less negativity and more neutrality. Sentiments by city showed most tweets in both periods coming from Riyadh and Jeddah in Saudi-Arabia, and from Salt in Jordan. Conclusions: Despite the limitations of the study, the change in sentiments detected by the study may suggest more vaccine acceptance which raises some hope and perhaps a higher vaccine uptake in the future. The sentiments by city findings may help in the design of future research.

ReDWINE: A clinical datamart with text analytical capabilities to facilitate rehabilitation research

Rehabilitation research focuses on determining the components of a treatment intervention, the mechanism of how these components lead to recovery and rehabilitation, and ultimately the optimal intervention strategies to maximize patients’ physical, psychologic, and social functioning. Traditional randomized clinical trials that study and establish new interventions face challenges, such as high cost and time commitment. Observational studies that use existing clinical data to observe the effect of an intervention have shown several advantages over RCTs. Electronic Health Records (EHRs) have become an increasingly important resource for conducting observational studies. To support these studies, we developed a clinical research datamart, called ReDWINE (Rehabilitation Datamart With Informatics iNfrastructure for rEsearch), that transforms the rehabilitation-related EHR data collected from the UPMC health care system to the Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) to facilitate rehabilitation research. The standardized EHR data stored in ReDWINE will further reduce the time and effort required by investigators to pool, harmonize, clean, and analyze data from multiple sources, leading to more robust and comprehensive research findings. ReDWINE also includes deployment of data visualization and data analytics tools to facilitate cohort definition and clinical data analysis. These include among others the Open Health Natural Language Processing (OHNLP) toolkit, a high-throughput NLP pipeline, to provide text analytical capabilities at scale in ReDWINE. Using this comprehensive representation of patient data in ReDWINE for rehabilitation research will facilitate real-world evidence for health interventions and outcomes.

Narratives in economics

AbstractThere is growing awareness within the economics profession of the important role narratives play in the economy. Even though empirical approaches that try to quantify economic narratives are getting increasingly popular, there is no theory or even a universally accepted definition of economic narratives underlying this research. First, we review and categorize the economic literature concerned with narratives and work out the different paradigms at play. Only a subset of the literature considers narratives to be active drivers of economic activity. To solidify the foundation of narrative economics, we propose a definition of collective economic narratives, isolating five important characteristics. We argue that, for a narrative to be economically relevant, it must be a sense‐making story that emerges in a social context and suggests action to a social group. We also systematize how a collective economic narrative differs from a topic and from other kinds of narratives that are likely to have less impact on the economy. With regard to the popular use of topic modeling, we suggest that the complementary use of other methods from the natural language processing (NLP) toolkit and the development of new methods is inevitable to go beyond identifying topics and move towards true empirical narrative economics.

Sentiment Analysis and Comprehensive Evaluation of Supervised Machine Learning Models Using Twitter Data on Russia-Ukraine War.

The Russia-Ukrainian War refers to the ongoing hostilities between Russia and Ukraine. It was first focused on whether Crimea and the Donbass were formally recognised as being a part of Ukraine when Russia started it in February 2014. The conflict dramatically grew when Russia began its incursion of Ukraine on February 24, 2022, following a military build-up on the Russian-Ukrainian border that started in late 2021. Examining public perceptions of the crisis between Russia and Ukraine is the goal of this piece. These days, social media has taken on a significant role in communication, and as a result, opinions can be found on platforms like Facebook, Twitter, and Instagram. The study makes use of his 11,250 tweets about the war between Russia and Ukraine from his Twitter account. Techniques, including image processing, object identification, and natural language processing, have shown application, power, and potential for machine learning. The same applies to text analytics. For text analysis, sentiment analysis, and entity annotation, machine learning techniques are frequently employed. According to the applicability and efficacy of the machine learning model, natural language processing toolkit in python is utilised in to examine the textual polarity and subjectivity score of tweets. Moreover, because machine learning models have a high degree of classification accuracy, they have been widely utilised to categorise emotions. We have developed and test models using three feature extraction techniques: TF-IDF (term frequency-inverse document frequency), BoW (bag of words), and N-gram. Each model was assessed using a number of important performance indicators, including accuracy, precision, recall, and F1 score. Results show that the extra trees classifier (ETC) model achieves a highest accuracy of 0.84 in combination with the Bow property which is a measure to evaluate the efficacy of a machine learning algorithm. Logistic regression (LR), decision tree (DT), support vector machine (SVM), XGB, Gaussian naive base (GNB), ADA, and K-nearest neighbours (KNN) comparison have also been made.

Agile Development Methodologies and Natural Language Processing: A Mapping Review

Agile software development is one of the most important development paradigms these days. However, there are still some challenges to consider to reduce problems during the documentation process. Some assistive methods have been created to support developers in their documentation activities. In this regard, Natural Language Processing (NLP) can be used to create various related tools (such as assistants) to help with the documentation process. This paper presents the current state-of-the-art NLP techniques used in the agile development documentation process. A mapping review was done to complete the objective, the search strategy is used to obtain relevant studies from ScienceDirect, IEEE Xplore, ACM Digital Library, SpringerLink, and Willey. The search results after inclusion and exclusion criteria application left 47 relevant papers identified. These papers were analyzed to obtain the most used NLP techniques and NLP toolkits. The toolkits were also classified by the kind of techniques that are available in each of them. In addition, the behavior of the research area over time was analyzed using the relevant paper found by year. We found that performance measuring methods are not standardized, and, in consequence, the works are not easily comparable. In general, the number of related works and its distribution per year shows a growing trend of the works related to this topic in recent years; this indicates that the adoption of NLP techniques to improve agile methodologies is increasing.

Ambient Assisted Living: Scoping Review of Artificial Intelligence Models, Domains, Technology, and Concerns.

Ambient assisted living (AAL) is a common name for various artificial intelligence (AI)-infused applications and platforms that support their users in need in multiple activities, from health to daily living. These systems use different approaches to learn about their users and make automated decisions, known as AI models, for personalizing their services and increasing outcomes. Given the numerous systems developed and deployed for people with different needs, health conditions, and dispositions toward the technology, it is critical to obtain clear and comprehensive insights concerning AI models used, along with their domains, technology, and concerns, to identify promising directions for future work. This study aimed to provide a scoping review of the literature on AI models in AAL. In particular, we analyzed specific AI models used in AАL systems, the target domains of the models, the technology using the models, and the major concerns from the end-user perspective. Our goal was to consolidate research on this topic and inform end users, health care professionals and providers, researchers, and practitioners in developing, deploying, and evaluating future intelligent AAL systems. This study was conducted as a scoping review to identify, analyze, and extract the relevant literature. It used a natural language processing toolkit to retrieve the article corpus for an efficient and comprehensive automated literature search. Relevant articles were then extracted from the corpus and analyzed manually. This review included 5 digital libraries: IEEE, PubMed, Springer, Elsevier, and MDPI. We included a total of 108 articles. The annual distribution of relevant articles showed a growing trend for all categories from January 2010 to July 2022. The AI models mainly used unsupervised and semisupervised approaches. The leading models are deep learning, natural language processing, instance-based learning, and clustering. Activity assistance and recognition were the most common target domains of the models. Ambient sensing, mobile technology, and robotic devices mainly implemented the models. Older adults were the primary beneficiaries, followed by patients and frail persons of various ages. Availability was a top beneficiary concern. This study presents the analytical evidence of AI models in AAL and their domains, technologies, beneficiaries, and concerns. Future research on intelligent AAL should involve health care professionals and caregivers as designers and users, comply with health-related regulations, improve transparency and privacy, integrate with health care technological infrastructure, explain their decisions to the users, and establish evaluation metrics and design guidelines. PROSPERO (International Prospective Register of Systematic Reviews) CRD42022347590; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022347590.

A thermoelectric materials database auto-generated from the scientific literature using ChemDataExtractor

An auto-generated thermoelectric-materials database is presented, containing 22,805 data records, automatically generated from the scientific literature, spanning 10,641 unique extracted chemical names. Each record contains a chemical entity and one of the seminal thermoelectric properties: thermoelectric figure of merit, ZT; thermal conductivity, κ; Seebeck coefficient, S; electrical conductivity, σ; power factor, PF; each linked to their corresponding recorded temperature, T. The database was auto-generated using the automatic sentence-parsing capabilities of the chemistry-aware, natural language processing toolkit, ChemDataExtractor 2.0, adapted for application in the thermoelectric-materials domain, following a rule-based sentence-simplification step. Data were mined from the text of 60,843 scientific papers that were sourced from three scientific publishers: Elsevier, the Royal Society of Chemistry, and Springer. To the best of our knowledge, this is the first automatically-generated database of thermoelectric materials and their properties from existing literature. The database was evaluated to have a precision of 82.25% and has been made publicly available to facilitate the application of data science in the thermoelectric-materials domain, for analysis, design, and prediction.

Utility of Features in a Natural-Language-Processing-Based Clinical De-Identification Model Using Radiology Reports for Advanced NSCLC Patients

The de-identification of clinical reports is essential to protect the confidentiality of patients. The natural-language-processing-based named entity recognition (NER) model is a widely used technique of automatic clinical de-identification. The performance of such a machine learning model relies largely on the proper selection of features. The objective of this study was to investigate the utility of various features in a conditional-random-field (CRF)-based NER model. Natural language processing (NLP) toolkits were used to annotate the protected health information (PHI) from a total of 10,239 radiology reports that were divided into seven types. Multiple features were extracted by the toolkit and the NER models were built using these features and their combinations. A total of 10 features were extracted and the performance of the models was evaluated based on their precision, recall, and F1-score. The best-performing features were n-gram, prefix-suffix, word embedding, and word shape. These features outperformed others across all types of reports. The dataset we used was large in volume and divided into multiple types of reports. Such a diverse dataset made sure that the results were not subject to a small number of structured texts from where a machine learning model can easily learn the features. The manual de-identification of large-scale clinical reports is impractical. This study helps to identify the best-performing features for building an NER model for automatic de-identification from a wide array of features mentioned in the literature.

Natural Language Processing of Radiology Reports to Detect Complications of Ischemic Stroke.

Abstraction of critical data from unstructured radiologic reports using natural language processing (NLP) is a powerful tool to automate the detection of important clinical features and enhance research efforts. We present a set of NLP approaches to identify critical findings in patients with acute ischemic stroke from radiology reports of computed tomography (CT) and magnetic resonance imaging (MRI). We trained machine learning classifiers to identify categorical outcomes of edema, midline shift (MLS), hemorrhagic transformation, and parenchymal hematoma, as well as rule-based systems (RBS) to identify intraventricular hemorrhage (IVH) and continuous MLS measurements within CT/MRI reports. Using a derivation cohort of 2289 reports from 550 individuals with acute middle cerebral artery territory ischemic strokes, we externally validated our models on reports from a separate institution as well as from patients with ischemic strokes in any vascular territory. In all data sets, a deep neural network with pretrained biomedical word embeddings (BioClinicalBERT) achieved the highest discrimination performance for binary prediction of edema (area under precision recall curve [AUPRC] > 0.94), MLS (AUPRC > 0.98), hemorrhagic conversion (AUPRC > 0.89), and parenchymal hematoma (AUPRC > 0.76). BioClinicalBERT outperformed lasso regression (p < 0.001) for all outcomes except parenchymal hematoma (p = 0.755). Tailored RBS for IVH and continuous MLS outperformed BioClinicalBERT (p < 0.001) and linear regression, respectively (p < 0.001). Our study demonstrates robust performance and external validity of a core NLP tool kit for identifying both categorical and continuous outcomes of ischemic stroke from unstructured radiographic text data. Medically tailored NLP methods have multiple important big data applications, including scalable electronic phenotyping, augmentation of clinical risk prediction models, and facilitation of automatic alert systems in the hospital setting.

An open-source natural language processing toolkit to support software development: addressing automatic bug detection, code summarisation and code search

This paper aims to introduce the innovative work carried out in the Horizon 2020 DECODER project – acronym for “DEveloper COmpanion for Documented and annotatEd code Reference” – (Grant Agreement no. 824231) by linking the fields of natural language processing (NLP) and software engineering. The project as a whole addresses the development of a framework, namely the Persistent Knowledge Monitor (PKM), that acts as a central infrastructure to store, access, and trace all the data, information and knowledge related to a given software or ecosystem. This meta-model defines the knowledge base that can be queried and analysed by all the tools integrated and developed in DECODER. Besides, the DECODER project offers a friendly user interface where each of the predefined three roles (i.e., developers, maintainers and reviewers) can access and query the PKM with their personal accounts. The paper focuses on the NLP tools developed and integrated in the PKM, namely the deep learning models developed to perform variable misuse, code summarisation and semantic parsing. These were developed under a common work package – “Activities for the developer” – intended to precisely target developers, who can perform tasks such as detection of bugs, automatic generation of documentation for source code and generation of code snippets from natural languages instructions, among the multiple functionalities that DECODER offers. These tools assist and help the developers in the daily work, by increasing their productivity and avoiding loss of time in tedious tasks such as manual bug detection. Training and validation were conducted for four use cases in Java, C and C++ programming languages in order to evaluate the performance, suitability, usability, etc. of the developed tools.

A survey on extraction of causal relations from natural language text

As an essential component of human cognition, cause–effect relations appear frequently in text, and curating cause–effect relations from text helps in building causal networks for predictive tasks. Existing causality extraction techniques include knowledge-based, statistical machine learning (ML)-based, and deep learning-based approaches. Each method has its advantages and weaknesses. For example, knowledge-based methods are understandable but require extensive manual domain knowledge and have poor cross-domain applicability. Statistical machine learning methods are more automated because of natural language processing (NLP) toolkits. However, feature engineering is labor-intensive, and toolkits may lead to error propagation. In the past few years, deep learning techniques attract substantial attention from NLP researchers because of its powerful representation learning ability and the rapid increase in computational resources. Their limitations include high computational costs and a lack of adequate annotated training data. In this paper, we conduct a comprehensive survey of causality extraction. We initially introduce primary forms existing in the causality extraction: explicit intra-sentential causality, implicit causality, and inter-sentential causality. Next, we list benchmark datasets and modeling assessment methods for causal relation extraction. Then, we present a structured overview of the three techniques with their representative systems. Lastly, we highlight existing open challenges with their potential directions.

The possibility of improving automated calculation of measures of lexical richness for EFL writing: A comparison of the LCA, NLTK and SpaCy tools

Automatically calculating measures of lexical richness is important for L2 learning because they can be used for assessment of productive abilities and general linguistic ability. One popular tool for doing so is the Lexical Complexity Analyzer (LCA), but more advanced tools for parsing have become available since its creation. This paper compares a modified version of the LCA code run with NLTK and SpaCy, two popular natural language processing toolkits, and the online version of the LCA to calculate 26 measures of lexical richness. We show how similarly they calculate the measures and how well each of the three tools' calculations correlate with EFL writer's human-rated scores and TOEFL® ITP scores. We found that six of the measures suggested to be associated with higher oral proficiency by Lu (2012) were also highly correlated with higher human-rated scores and TOEFL® ITP scores in our data set. However, the modifications to our code that utilize a different list to determine word sophistication and allow be and have verbs to be treated as lexical verbs caused four measures which Lu (2012) found to be unassociated with proficiency to be correlated with both human-rated scores and TOEFL® ITP scores, particularly when run with SpaCy.

VOD and OTT Content as a Therapeutic Measure to Improve Self-esteem: A Retrospective Analysis of user Generated Opinion using NLP Techniques

Objectives: [a] To understand how popular millennial media usage practices can be used to improve self-esteem among the millennial cohort; [b] To ascertain the effect of VoD and OTT streaming services in improving levels of self-esteem. Methods: Natural Language Processing techniques, namely, Term Frequency analysis, Network analysis, and Opinion Mining were used to identify and analyze the user generated Twitter Corpus dataset. Using Tweepy, a Python module, the data (n = 2566 tweets) were retrieved from the RESTAPI (Application Programming Interface) of Twitter. NLTK (Natural Language Processing ToolKit), NetworkX, and TextBlob were used for Opinion mining and visualization of the datasets Findings: Binge-watching of VoD and OTT content that address issues related to Body Positivity and Self-esteem resulted in improvement among the viewer cohort. Analysis of the Twitter corpus dataset implied that the following latent factors contributed toward the observed outcome: relatability (23:14%), upliftment(19.4%), gratitude(36.11%), and relief (21.2%). Novelty: Application of Natural Language Processing to identify latent factors that contribute toward either positive or negative outcome(s) has not been conducted before. Our previous research on similar areas identified latent factors that contribute toward behavioral outcomes in terms of attitude toward vaccines. Keywords: Selfesteem; Bingewatching; Therapy; Netflix; Behavioral outcomes

Investigation of the Utility of Features in a Clinical De-identification Model: A Demonstration Using EHR Pathology Reports for Advanced NSCLC Patients.

BackgroundElectronic health record (EHR) systems contain a large volume of texts, including visit notes, discharge summaries, and various reports. To protect the confidentiality of patients, these records often need to be fully de-identified before circulating for secondary use. Machine learning (ML) based named entity recognition (NER) model has emerged as a popular technique of automatic de-identification.ObjectiveThe performance of a machine learning model highly depends on the selection of appropriate features. The objective of this study was to investigate the usability of multiple features in building a conditional random field (CRF) based clinical de-identification NER model.MethodsUsing open-source natural language processing (NLP) toolkits, we annotated protected health information (PHI) in 1,500 pathology reports and built supervised NER models using multiple features and their combinations. We further investigated the dependency of a model's performance on the size of training data.ResultsAmong the 10 feature extractors explored in this study, n-gram, prefix–suffix, word embedding, and word shape performed the best. A model using combination of these four feature sets yielded precision, recall, and F1-score for each PHI as follows: NAME (0.80; 0.79; 0.80), LOCATION (0.85; 0.83; 0.84), DATE (0.86; 0.79; 0.82), HOSPITAL (0.96; 0.93; 0.95), ID (0.99; 0.82; 0.90), and INITIALS (0.97; 0.49; 0.65). We also found that the model's performance becomes saturated when the training data size is beyond 200.ConclusionManual de-identification of large-scale data is an impractical procedure since it is time-consuming and subject to human errors. Analysis of the NER model's performance in this study sheds light on a semi-automatic clinical de-identification pipeline for enterprise-wide data warehousing.

UNLT: Urdu Natural Language Toolkit

AbstractThis study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of theUNLT (Urdu Natural Language Toolkit)which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochasticn-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.