Architecture For Natural Language Processing Research Articles

Using Transformer-Based Topic Modeling to Examine Discussions of Delta-8 Tetrahydrocannabinol: Content Analysis.

Delta-8 tetrahydrocannabinol (THC) is a psychoactive cannabinoid found in small amounts naturally in the cannabis plant; it can also be synthetically produced in larger quantities from hemp-derived cannabidiol. Most states permit the sale of hemp and hemp-derived cannabidiol products; thus, hemp-derived delta-8 THC products have become widely available in many state hemp marketplaces, even where delta-9 THC, the most prominently occurring THC isomer in cannabis, is not currently legal. Health concerns related to the processing of delta-8 THC products and their psychoactive effects remain understudied. The goal of this study is to implement a novel topic modeling approach based on transformers, a state-of-the-art natural language processing architecture, to identify and describe emerging trends and topics of discussion about delta-8 THC from social media discourse, including potential symptoms and adverse health outcomes experienced by people using delta-8 THC products. Posts from January 2008 to December 2021 discussing delta-8 THC were isolated from cannabis-related drug forums on Reddit (Reddit Inc), a social media platform that hosts the largest web-based drug forums worldwide. Unsupervised topic modeling with state-of-the-art transformer-based models was used to cluster posts into topics and assign labels describing the kinds of issues being discussed with respect to delta-8 THC. Results were then validated by human subject matter experts. There were 41,191 delta-8 THC posts identified and 81 topics isolated, the most prevalent being (1) discussion of specific brands or products, (2) comparison of delta-8 THC to other hemp-derived cannabinoids, and (3) safety warnings. About 5% (n=1220) of posts from the resulting topics included content discussing health-related symptoms such as anxiety, sleep disturbance, and breathing problems. Until 2020, Reddit posts contained fewer than 10 mentions of delta-8-THC for every 100,000 cannabis posts annually. However, in 2020, these rates increased by 13 times the 2019 rate (to 99.2 mentions per 100,000 cannabis posts) and continued to increase into 2021 (349.5 mentions per 100,000 cannabis posts). Our study provides insights into emerging public health concerns around delta-8 THC, a novel substance about which little is known. Furthermore, we demonstrate the use of transformer-based unsupervised learning approaches to derive intelligible topics from highly unstructured discussions of delta-8 THC, which may help improve the timeliness of identification of emerging health concerns related to new substances.

Learning the language of QCD jets with transformers

Transformers have become the primary architecture for natural language processing. In this study, we explore their use for auto-regressive density estimation in high-energy jet physics, which involves working with a high-dimensional space. We draw an analogy between sentences and words in natural language and jets and their constituents in high-energy physics. Specifically, we investigate density estimation for light QCD jets and hadronically decaying boosted top jets. Since transformers allow easy sampling from learned densities, we exploit their generative capability to assess the quality of the density estimate. Our results indicate that the generated data samples closely resemble the original data, as evidenced by the excellent agreement of distributions such as particle multiplicity or jet mass. Furthermore, the generated samples are difficult to distinguish from the original data, even by a powerful supervised classifier. Given their exceptional data processing capabilities, transformers could potentially be trained directly on the massive LHC data sets to learn the probability densities in high-energy jet physics.

Categorical representation learning and RG flow operators for algorithmic classifiers

Following the earlier formalism of the categorical representation learning, we discuss the construction of the ‘RG-flow-based categorifier’. Borrowing ideas from the theory of renormalization group (RG) flows in quantum field theory, holographic duality, and hyperbolic geometry and combining them with neural ordinary differential equation techniques, we construct a new algorithmic natural language processing architecture, called the RG-flow categorifier or for short the RG categorifier, which is capable of data classification and generation in all layers. We apply our algorithmic platform to biomedical data sets and show its performance in the field of sequence-to-function mapping. In particular, we apply the RG categorifier to particular genomic sequences of flu viruses and show how our technology is capable of extracting the information from given genomic sequences, finding their hidden symmetries and dominant features, classifying them, and using the trained data to make a stochastic prediction of new plausible generated sequences associated with a new set of viruses which could avoid the human immune system.

PMACNet: Parallel Multiscale Attention Constraint Network for Pan-Sharpening

Pan-sharpening, a task involving information fusion, entails merging panchromatic (PAN) images with high spatial resolution and low-resolution multispectral (LRMS) images in order to obtain high-resolution multispectral (HRMS) images. Due to deep learning’s excellent regression capabilities, it has recently become the dominating technique for this assignment. Meanwhile, the development of the transformer, a novel deep learning architecture for natural language processing, has provided researchers with new insights. In this letter, we seek to extend transformer’s excellent mechanisms to pixel-level fusion challenges. We designed a parallel convolutional neural network structure for learning both the regions of interest from the LRMS images and the residuals required for regression to HRMS images. Then, in our proposed pixelwise attention constraint (PAC) module, the residuals will be changed utilizing the learned region of interest. In addition, we presented a novel multireceptive-field attention block (MRFAB) to frame our network. Experiments on two datasets also show that our work is better than the mainstream algorithms at both indicators and visualization.

To Batch or Not to Batch? Comparing Batching and Curriculum Learning Strategies across Tasks and Datasets

Many natural language processing architectures are greatly affected by seemingly small design decisions, such as batching and curriculum learning (how the training data are ordered during training). In order to better understand the impact of these decisions, we present a systematic analysis of different curriculum learning strategies and different batching strategies. We consider multiple datasets for three tasks: text classification, sentence and phrase similarity, and part-of-speech tagging. Our experiments demonstrate that certain curriculum learning and batching decisions do increase performance substantially for some tasks.

Prediction of chemical reaction yields using deep learning

Artificial intelligence is driving one of the most important revolutions in organic chemistry. Multiple platforms, including tools for reaction prediction and synthesis planning based on machine learning, have successfully become part of the organic chemists’ daily laboratory, assisting in domain-specific synthetic problems. Unlike reaction prediction and retrosynthetic models, the prediction of reaction yields has received less attention in spite of the enormous potential of accurately predicting reaction conversion rates. Reaction yields models, describing the percentage of the reactants converted to the desired products, could guide chemists and help them select high-yielding reactions and score synthesis routes, reducing the number of attempts. So far, yield predictions have been predominantly performed for high-throughput experiments using a categorical (one-hot) encoding of reactants, concatenated molecular fingerprints, or computed chemical descriptors. Here, we extend the application of natural language processing architectures to predict reaction properties given a text-based representation of the reaction, using an encoder transformer model combined with a regression layer. We demonstrate outstanding prediction performance on two high-throughput experiment reactions sets. An analysis of the yields reported in the open-source USPTO data set shows that their distribution differs depending on the mass scale, limiting the data set applicability in reaction yields predictions.

The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures

In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Transformer architecture, leading to designs such as BERT, GPT (I, II, III), etc. Although these large-size models have achieved unprecedented performances, they come at high computational costs. Consequently, some of the recent NLP architectures have utilized concepts of transfer learning, pruning, quantization, and knowledge distillation to achieve moderate model sizes while keeping nearly similar performances as achieved by their predecessors. Additionally, to mitigate the data size challenge raised by language models from a knowledge extraction perspective, Knowledge Retrievers have been built to extricate explicit data documents from a large corpus of databases with greater efficiency and accuracy. Recent research has also focused on superior inference by providing efficient attention to longer input sequences. In this paper, we summarize and examine the current state-of-the-art (SOTA) NLP models that have been employed for numerous NLP tasks for optimal performance and efficiency. We provide a detailed understanding and functioning of the different architectures, a taxonomy of NLP designs, comparative evaluations, and future directions in NLP.

The relational processing limits of classic and contemporary neural network models of language processing

ABSTRACT Whether neural networks can capture relational knowledge is a matter of long-standing controversy. Recently, some researchers have argued that (1) classic connectionist models can handle relational structure and (2) the success of deep learning approaches to natural language processing suggests that structured representations are unnecessary to model human language. We tested the Story Gestalt model, a classic connectionist model of text comprehension, and a Sequence-to-Sequence with Attention model, a modern deep learning architecture for natural language processing. Both models were trained to answer questions about stories based on abstract thematic roles. Two simulations varied the statistical structure of new stories while keeping their relational structure intact. The performance of each model fell below chance at least under one manipulation. We argue that both models fail our tests because they can't perform dynamic binding. These results cast doubts on the suitability of traditional neural networks for explaining relational reasoning and language processing phenomena.

Towards automated generation of curated datasets in radiology: Application of natural language processing to unstructured reports exemplified on CT for pulmonary embolism

PurposeTo design and evaluate a self-trainable natural language processing (NLP)-based procedure to classify unstructured radiology reports. The method enabling the generation of curated datasets is exemplified on CT pulmonary angiogram (CTPA) reports. MethodWe extracted the impressions of CTPA reports created at our institution from 2016 to 2018 (n = 4397; language: German). The status (pulmonary embolism: yes/no) was manually labelled for all exams. Data from 2016/2017 (n = 2801) served as a ground truth to train three NLP architectures that only require a subset of reference datasets for training to be operative. The three architectures were as follows: a convolutional neural network (CNN), a support vector machine (SVM) and a random forest (RF) classifier. Impressions of 2018 (n = 1377) were kept aside and used for general performance measurements. Furthermore, we investigated the dependence of classification performance on the amount of training data with multiple simulations. ResultsThe classification performance of all three models was excellent (accuracies: 97 %–99 %; F1 scores 0.88–0.97; AUCs: 0.993–0.997). Highest accuracy was reached by the CNN with 99.1 % (95 % CI 98.5–99.6 %). Training with 470 labelled impressions was sufficient to reach an accuracy of > 93 % with all three NLP architectures. ConclusionOur NLP-based approaches allow for an automated and highly accurate retrospective classification of CTPA reports with manageable effort solely using unstructured impression sections. We demonstrated that this approach is useful for the classification of radiology reports not written in English. Moreover, excellent classification performance is achieved at relatively small training set sizes.

Development and application of a high throughput natural language processing architecture to convert all clinical documents in a clinical data warehouse into standardized medical vocabularies.

Natural language processing (NLP) engines such as the clinical Text Analysis and Knowledge Extraction System are a solution for processing notes for research, but optimizing their performance for a clinical data warehouse remains a challenge. We aim to develop a high throughput NLP architecture using the clinical Text Analysis and Knowledge Extraction System and present a predictive model use case. The CDW was comprised of 1 103 038 patients across 10 years. The architecture was constructed using the Hadoop data repository for source data and 3 large-scale symmetric processing servers for NLP. Each named entity mention in a clinical document was mapped to the Unified Medical Language System concept unique identifier (CUI). The NLP architecture processed 83 867 802 clinical documents in 13.33 days and produced 37 721 886 606 CUIs across 8 standardized medical vocabularies. Performance of the architecture exceeded 500 000 documents per hour across 30 parallel instances of the clinical Text Analysis and Knowledge Extraction System including 10 instances dedicated to documents greater than 20 000 bytes. In a use-case example for predicting 30-day hospital readmission, a CUI-based model had similar discrimination to n-grams with an area under the curve receiver operating characteristic of 0.75 (95% CI, 0.74-0.76). Our health system's high throughput NLP architecture may serve as a benchmark for large-scale clinical research using a CUI-based approach.

Meaning-based machine learning for information assurance

Abstract This paper presents meaning-based machine learning, the use of semantically meaningful input data into machine learning systems in order to produce output that is meaningful to a human user where the semantic input comes from the Ontological Semantics Technology theory of natural language processing. How to bridge from knowledge-based natural language processing architectures to traditional machine learning systems is described to include high-level descriptions of the steps taken. These meaning-based machine learning systems are then applied to problems in information assurance and security that remain unsolved and feature large amounts of natural language text.

Little Things With Big Effects

Error diagnosis in ICALL typically analyzes learner input in an attempt to abstract and identify indicators of the learner's (mis)conceptions of linguistic properties. For written input, this process usually starts with the identification of tokens that will serve as the atomic building blocks of the analysis. In this paper, we discuss the consequences of mismatches between the learner's perception of a given token and the system's interpretation of its linguistic properties. Based on our analysis of the interaction of beginning learners of Portuguese with the ICALL system TAGARELA, we discuss why tokenization and the interpretation of accented characters deserve particular attention in a system used by language learners. On the computational side, we argue that the mismatches arising in such cases can be addressed in a general way by building ICALL systems on an annotation-based natural language processing architecture which monotonically enriches the representation of learner input.

Natural language processing: an introduction

To provide an overview and tutorial of natural language processing (NLP) and modern NLP-system design. This tutorial targets the medical informatics generalist who has limited acquaintance with the principles behind NLP and/or limited knowledge of the current state of the art. We describe the historical evolution of NLP, and summarize common NLP sub-problems in this extensive field. We then provide a synopsis of selected highlights of medical NLP efforts. After providing a brief description of common machine-learning approaches that are being used for diverse NLP sub-problems, we discuss how modern NLP architectures are designed, with a summary of the Apache Foundation's Unstructured Information Management Architecture. We finally consider possible future directions for NLP, and reflect on the possible impact of IBM Watson on the medical field.

Analyzing learner language: towards a flexible natural language processing architecture for intelligent language tutors

Intelligent language tutoring systems (ILTS) typically analyze learner input to diagnose learner language properties and provide individualized feedback. Despite a long history of ILTS research, such systems are virtually absent from real-life foreign language teaching (FLT). Taking a step toward more closely linking ILTS research to real-life FLT, in this article we investigate the connection between FLT activity design and the system architecture of an ILT system. We argue that a demand-driven, annotation-based natural language processing (NLP) architecture is well-suited to handle the demands posed by the heterogeneous learner input which results when supporting a wider range of FLT activity types. We illustrate how the unstructured information management architecture (UIMA) can be used in an ILTS, thereby connecting the specific needs of activities in foreign language teaching to the current research and development of NLP architectures in general. Making the conceptual issues concrete, we discuss the design and realization of a UIMA-based reimplementation of the NLP in the TAGARELA system, an intelligent web-based tutoring system supporting the teaching and learning of Portuguese.