Architecture Of Language Processing Research Articles

Using Transformer-Based Topic Modeling to Examine Discussions of Delta-8 Tetrahydrocannabinol: Content Analysis.

Delta-8 tetrahydrocannabinol (THC) is a psychoactive cannabinoid found in small amounts naturally in the cannabis plant; it can also be synthetically produced in larger quantities from hemp-derived cannabidiol. Most states permit the sale of hemp and hemp-derived cannabidiol products; thus, hemp-derived delta-8 THC products have become widely available in many state hemp marketplaces, even where delta-9 THC, the most prominently occurring THC isomer in cannabis, is not currently legal. Health concerns related to the processing of delta-8 THC products and their psychoactive effects remain understudied. The goal of this study is to implement a novel topic modeling approach based on transformers, a state-of-the-art natural language processing architecture, to identify and describe emerging trends and topics of discussion about delta-8 THC from social media discourse, including potential symptoms and adverse health outcomes experienced by people using delta-8 THC products. Posts from January 2008 to December 2021 discussing delta-8 THC were isolated from cannabis-related drug forums on Reddit (Reddit Inc), a social media platform that hosts the largest web-based drug forums worldwide. Unsupervised topic modeling with state-of-the-art transformer-based models was used to cluster posts into topics and assign labels describing the kinds of issues being discussed with respect to delta-8 THC. Results were then validated by human subject matter experts. There were 41,191 delta-8 THC posts identified and 81 topics isolated, the most prevalent being (1) discussion of specific brands or products, (2) comparison of delta-8 THC to other hemp-derived cannabinoids, and (3) safety warnings. About 5% (n=1220) of posts from the resulting topics included content discussing health-related symptoms such as anxiety, sleep disturbance, and breathing problems. Until 2020, Reddit posts contained fewer than 10 mentions of delta-8-THC for every 100,000 cannabis posts annually. However, in 2020, these rates increased by 13 times the 2019 rate (to 99.2 mentions per 100,000 cannabis posts) and continued to increase into 2021 (349.5 mentions per 100,000 cannabis posts). Our study provides insights into emerging public health concerns around delta-8 THC, a novel substance about which little is known. Furthermore, we demonstrate the use of transformer-based unsupervised learning approaches to derive intelligible topics from highly unstructured discussions of delta-8 THC, which may help improve the timeliness of identification of emerging health concerns related to new substances.

Categorical representation learning and RG flow operators for algorithmic classifiers

Following the earlier formalism of the categorical representation learning, we discuss the construction of the ‘RG-flow-based categorifier’. Borrowing ideas from the theory of renormalization group (RG) flows in quantum field theory, holographic duality, and hyperbolic geometry and combining them with neural ordinary differential equation techniques, we construct a new algorithmic natural language processing architecture, called the RG-flow categorifier or for short the RG categorifier, which is capable of data classification and generation in all layers. We apply our algorithmic platform to biomedical data sets and show its performance in the field of sequence-to-function mapping. In particular, we apply the RG categorifier to particular genomic sequences of flu viruses and show how our technology is capable of extracting the information from given genomic sequences, finding their hidden symmetries and dominant features, classifying them, and using the trained data to make a stochastic prediction of new plausible generated sequences associated with a new set of viruses which could avoid the human immune system.

A Developmental Approach to Assessing and Treating Agrammatic Aphasia.

There is mounting evidence that the agrammatism that defines Broca's aphasia can be explained in processing terms. However, the extant approach simply describes agrammatism as disparate deficits in a static, mature system. This tutorial aims to motivate and outline a developmental alternative. This alternative is processability theory (PT), a root-to-apex theory of language development, with its origins in the field of second language acquisition, which can connect the findings of aphasia research. This tutorial critically reviews research on agrammatism as a language deficit, a representational deficit, and a processing phenomenon. Given evidence from research applying PT to language disorders, this tutorial outlines PT's multidimensional architecture of language processing. Using an emergence (onset) criterion, PT predicts fixed developmental stages in word order (syntax) and inflection (morphology) and individual differences in the timing of syntax and morphology. To link PT to agrammatism, this theory's applications to diagnosis and teaching are overviewed, and a case study of five individuals with moderate agrammatism is presented. Analysis showed that all individuals were positioned in the early PT stages and differed in their timing of syntax and morphology consistent with theoretical predictions. Evidence from the case study suggests that, although agrammatism results from neural damage and associated language loss, the processing procedures necessary for relearning remain and can be exploited for recovery. A program of diagnosis and intervention is proposed, and future research directions are discussed. https://doi.org/10.23641/asha.19416488.

SignNet II: A Transformer-Based Two-Way Sign Language Translation Model.

The role of a sign interpreting agent is to bridge the communication gap between the hearing-only and Deaf or Hard of Hearing communities by translating both from sign language to text and from text to sign language. Until now, much of the AI work in automated sign language processing has focused primarily on sign language to text translation, which puts the advantage mainly on the side of hearing individuals. In this work, we describe advances in sign language processing based on transformer networks. Specifically, we introduce SignNet II, a sign language processing architecture, a promising step towards facilitating two-way sign language communication. It is comprised of sign-to-text and text-to-sign networks jointly trained using a dual learning mechanism. Furthermore, by exploiting the notion of sign similarity, a metric embedding learning process is introduced to enhance the text-to-sign translation performance. Using a bank of multi-feature transformers, we analyzed several input feature representations and discovered that keypoint-based pose features consistently performed well, irrespective of the quality of the input videos. We demonstrated that the two jointly trained networks outperformed their singly-trained counterparts, showing noteworthy enhancements in BLEU-1 - BLEU-4 scores when tested on the largest available German Sign Language (GSL) benchmark dataset.

Compositionality in a Parallel Architecture for Language Processing.

Compositionality has been a central concept in linguistics and philosophy for decades, and it is increasingly prominent in many other areas of cognitive science. Its status, however, remains contentious. Here, I reassess the nature and scope of the principle of compositionality (Partee, 1995) from the perspective of psycholinguistics and cognitive neuroscience. First, I review classic arguments for compositionality and conclude that they fail to establish compositionality as a property of human language. Next, I state a new competence argument, acknowledging the fact that any competent user of a language L can assign to most expressions in L at least one meaning which is a function only of the meanings of the expression's parts and of its syntactic structure. I then discuss selected results from cognitive neuroscience, indicating that the human brain possesses the processing capacities presupposed by the competence argument. Finally, I outline a language processing architecture consistent with the neuroscience results, where semantic representations may be generated by a syntax-driven stream and by an "asyntactic" processing stream, jointly or independently. Compositionality is viewed as a constraint on computation in the former stream only.

Development and application of a high throughput natural language processing architecture to convert all clinical documents in a clinical data warehouse into standardized medical vocabularies.

Natural language processing (NLP) engines such as the clinical Text Analysis and Knowledge Extraction System are a solution for processing notes for research, but optimizing their performance for a clinical data warehouse remains a challenge. We aim to develop a high throughput NLP architecture using the clinical Text Analysis and Knowledge Extraction System and present a predictive model use case. The CDW was comprised of 1 103 038 patients across 10 years. The architecture was constructed using the Hadoop data repository for source data and 3 large-scale symmetric processing servers for NLP. Each named entity mention in a clinical document was mapped to the Unified Medical Language System concept unique identifier (CUI). The NLP architecture processed 83 867 802 clinical documents in 13.33 days and produced 37 721 886 606 CUIs across 8 standardized medical vocabularies. Performance of the architecture exceeded 500 000 documents per hour across 30 parallel instances of the clinical Text Analysis and Knowledge Extraction System including 10 instances dedicated to documents greater than 20 000 bytes. In a use-case example for predicting 30-day hospital readmission, a CUI-based model had similar discrimination to n-grams with an area under the curve receiver operating characteristic of 0.75 (95% CI, 0.74-0.76). Our health system's high throughput NLP architecture may serve as a benchmark for large-scale clinical research using a CUI-based approach.

The Long View of Language Localization.

For British neurologists, one case was considered to represent significant evidence regarding the organization of language in the brain in the second half of the 19th century. The interpretation of its significance was based on repeated standard clinical assessment of behavioral deficits, the use of a psychological model of processing, and lesion localization to inform understanding of clinic-pathological correlation. The aphasic deficits experienced by a single case were observed and recorded by London neurologist Henry Charlton Bastian (1837–1915) over a period of 18 years and used as a demonstration of clinico-pathological reasoning regarding language function. This case was well documented in many of Bastian’s publications; presented in teaching demonstrations; included in discussions at medical society meetings and public lectures; and reported widely in the medical press. When this patient died, the autopsy findings were added to the extensive record of his language deficits. Some aspects of the size and site of the lesion were consistent with Bastian’s clinical predictions arising from his model of language processing, while others presented more of a paradox. This single case was a significant source of discussion and reflection in the medical community throughout the second half of the 19th century. Examination of various interpretations of this case reveal the assumptions regarding the functional architecture of language processing and more general theoretical considerations of how evidence from cases of acquired neurogenic aphasia can be employed in developing such models. This long view into a historical case sheds light on the challenges of clinic-pathological correlation methods in the understanding of localization of language functions which remain today.

Remaining Puzzles about Morpheme Production in the Posterior Temporal Lobe

Using data from time-resolved cortical stimulation, intracranial neural recordings, and focal surgical resections, Lee et al. (2018) demonstrate that a small area within left posterior superior temporal gyrus (pSTG) supports the ability to produce functional morphemes but not other basic aspects of language production or comprehension. These findings are intriguing because they raise important questions about the functional architecture of language processing, including critically, the relationship between production and comprehension. Here, we highlight some of the puzzles that remain and that we hope will guide future empirical explorations of the cognitive and neural mechanisms that support our capacity for language.

Retrieval cues and syntactic ambiguity resolution: speed-accuracy tradeoff evidence

ABSTRACTLanguage comprehension involves coping with ambiguity and recovering from misanalysis. Syntactic ambiguity resolution is associated with increased reading times, a classic finding that has shaped theories of sentence processing. However, reaction times conflate the time it takes a process to complete with the quality of the behavior-related information available to the system. We therefore used the speed-accuracy tradeoff procedure (SAT) to derive orthogonal estimates of processing time and interpretation accuracy, and tested whether stronger retrieval cues (via semantic relatedness: neighed->horse vs. fell->horse) aid interpretation during recovery. On average, ambiguous sentences took 250ms longer (SAT rate) to interpret than unambiguous controls, demonstrating veridical differences in processing time. Retrieval cues more strongly related to the true subject always increased accuracy, regardless of ambiguity. These findings are consistent with a language processing architecture where cue-driven operations give rise to interpretation, and wherein diagnostic cues aid retrieval, regardless of parsing difficulty or structural uncertainty.

Neural organization of spoken language revealed by lesion-symptom mapping.

Studies of patients with acquired cognitive deficits following brain damage and studies using contemporary neuroimaging techniques form two distinct streams of research on the neural basis of cognition. In this study, we combine high-quality structural neuroimaging analysis techniques and extensive behavioral assessment of patients with persistent acquired language deficits to study the neural basis of language. Our results reveal two major divisions within the language system – meaning vs. form and recognition vs. production – and their instantiation in the brain. Phonological form deficits are associated with lesions in peri-Sylvian regions, whereas semantic production and recognition deficits are associated with damage to the left anterior temporal lobe and white matter connectivity with frontal cortex, respectively. These findings provide a novel synthesis of traditional and contemporary views of the cognitive and neural architecture of language processing, emphasizing dual-routes for speech processing and convergence of white matter tracts for semantic control and/or integration.

The ins and outs of meaning: Behavioral and neuroanatomical dissociation of semantically-driven word retrieval and multimodal semantic recognition in aphasia

Theories about the architecture of language processing differ with regard to whether verbal and nonverbal comprehension share a functional and neural substrate and how meaning extraction in comprehension relates to the ability to use meaning to drive verbal production. We (re-)evaluate data from 17 cognitive-linguistic performance measures of 99 participants with chronic aphasia using factor analysis to establish functional components and support vector regression-based lesion-symptom mapping to determine the neural correlates of deficits on these functional components. The results are highly consistent with our previous findings: production of semantic errors is behaviorally and neuroanatomically distinct from verbal and nonverbal comprehension. Semantic errors were most strongly associated with left ATL damage whereas deficits on tests of verbal and non-verbal semantic recognition were most strongly associated with damage to deep white matter underlying the frontal lobe at the confluence of multiple tracts, including the inferior fronto-occipital fasciculus, the uncinate fasciculus, and the anterior thalamic radiations. These results suggest that traditional views based on grey matter hub(s) for semantic processing are incomplete and that the role of white matter in semantic cognition has been underappreciated.

Little Things With Big Effects

Error diagnosis in ICALL typically analyzes learner input in an attempt to abstract and identify indicators of the learner's (mis)conceptions of linguistic properties. For written input, this process usually starts with the identification of tokens that will serve as the atomic building blocks of the analysis. In this paper, we discuss the consequences of mismatches between the learner's perception of a given token and the system's interpretation of its linguistic properties. Based on our analysis of the interaction of beginning learners of Portuguese with the ICALL system TAGARELA, we discuss why tokenization and the interpretation of accented characters deserve particular attention in a system used by language learners. On the computational side, we argue that the mismatches arising in such cases can be addressed in a general way by building ICALL systems on an annotation-based natural language processing architecture which monotonically enriches the representation of learner input.

Analyzing learner language: towards a flexible natural language processing architecture for intelligent language tutors

Intelligent language tutoring systems (ILTS) typically analyze learner input to diagnose learner language properties and provide individualized feedback. Despite a long history of ILTS research, such systems are virtually absent from real-life foreign language teaching (FLT). Taking a step toward more closely linking ILTS research to real-life FLT, in this article we investigate the connection between FLT activity design and the system architecture of an ILT system. We argue that a demand-driven, annotation-based natural language processing (NLP) architecture is well-suited to handle the demands posed by the heterogeneous learner input which results when supporting a wider range of FLT activity types. We illustrate how the unstructured information management architecture (UIMA) can be used in an ILTS, thereby connecting the specific needs of activities in foreign language teaching to the current research and development of NLP architectures in general. Making the conceptual issues concrete, we discuss the design and realization of a UIMA-based reimplementation of the NLP in the TAGARELA system, an intelligent web-based tutoring system supporting the teaching and learning of Portuguese.

Partner-specific adaptation in dialog.

No one denies that people adapt what they say and how they interpret what is said to them, depending on their interactive partners. What is controversial is when and how they do so. Several psycholinguistics research programs have found what appear to be failures to adapt to partners in the early moments of processing and have used this evidence to argue for modularity in the language processing architecture, claiming that the system cannot take into account a partner's distinct needs or knowledge early in processing. We review the evidence for both early and delayed partner-specific adaptations, and we identify some challenges and difficulties with interpreting this evidence. We then discuss new analyses from a previously published referential communication experiment (Metzing & Brennan, 2003) demonstrating that partner-specific effects need not occur late in processing. In contrast to Pickering and Garrod (2004) and Keysar, Barr, and Horton (1998b), we conclude that there is no good evidence that early processing has to be be "egocentric,""dumb," or encapsulated from social knowledge or common ground, but that under some circumstances, such as when one partner has made an attribution about another's knowledge or needs, processing can be nimble enough to adapt quite early to a perspective different from one's own.

Learning of control in a neural architecture of grounded language processing

Human cognition is characterized by three important features: productivity, dynamics and grounding. These features can be integrated in a neural architecture of language processing. The representations in this architecture always remain “in situ”, because they are grounded in perception, action, emotion, associations and (semantic) relations. The neural architecture shows how these representations can be combined in a productive manner, and how dynamics influences this process. The constraints that each of these features impose on each other result in an architecture in which local and global aspects interact in processing and learning. The architecture consists of neural “binding” mechanisms that produce (novel) sentence structures on the fly. Here, we discuss how the control of this binding process can be learned. We trained a feedforward network (FFN) for this task. The results show that information from the architecture is needed as input to learn control of binding. Thus, the control system is recurrent. We show that this recurrent system can learn control of binding for basic (but recursive) sentence structures. After learning, the binding process behaves well on a series of test sentences, including sentences with (unlimited) embeddings. However, for some of these sentences, difficulties arise due to dynamical binding conflicts in the architecture. We also discuss and illustrate the potential influence that the dynamics in the architecture could have on the binding process.

Should Psychology Ignore the Language of the Brain?

Claims that neuroscientific data do not contribute to our understanding of psychological functions have been made recently. Here I argue that these criticisms are solely based on an analysis of functional magnetic resonance imaging (fMRI) studies. However, fMRI is only one of the methods in the toolkit of cognitive neuroscience. I provide examples from research on event-related brain potentials (ERPs) that have contributed to our understanding of the cognitive architecture of human language functions. In addition, I provide evidence of (possible) contributions from fMRI measurements to our understanding of the functional architecture of language processing. Finally, I argue that a neurobiology of human language that integrates information about the necessary genetic and neural infrastructures will allow us to answer certain questions that are not answerable if all we have is evidence from behavior.

Brain correlates of discourse processing: An fMRI investigation of irony and conventional metaphor comprehension

Higher levels of discourse processing evoke patterns of cognition and brain activation that extend beyond the literal comprehension of sentences. We used fMRI to examine brain activation patterns while 16 healthy participants read brief three-sentence stories that concluded with either a literal, metaphoric, or ironic sentence. The fMRI images acquired during the reading of the critical sentence revealed a selective response of the brain to the two types of nonliteral utterances. Metaphoric utterances resulted in significantly higher levels of activation in the left inferior frontal gyrus and in bilateral inferior temporal cortex than the literal and ironic utterances. Ironic statements resulted in significantly higher activation levels than literal statements in the right superior and middle temporal gyri, with metaphoric statements resulting in intermediate levels in these regions. The findings show differential hemispheric sensitivity to these aspects of figurative language, and are relevant to models of the functional cortical architecture of language processing in connected discourse.

How the brain solves the binding problem for language: a neurocomputational model of syntactic processing

Syntax is one of the components in the architecture of language processing that allows the listener/reader to bind single-word information into a unified interpretation of multiword utterances. This paper discusses ERP effects that have been observed in relation to syntactic processing. The fact that these effects differ from the semantic N400 indicates that the brain honors the distinction between semantic and syntactic binding operations. Two models of syntactic processing attempt to account for syntax-related ERP effects. One type of model is serial, with a first phase that is purely syntactic in nature (syntax-first model). The other type of model is parallel and assumes that information immediately guides the interpretation process once it becomes available. This is referred to as the immediacy model. ERP evidence is presented in support of the latter model. Next, an explicit computational model is proposed to explain the ERP data. This Unification Model assumes that syntactic frames are stored in memory and retrieved on the basis of the spoken or written word form input. The syntactic frames associated with the individual lexical items are unified by a dynamic binding process into a structural representation that spans the whole utterance. On the basis of a meta-analysis of imaging studies on syntax, it is argued that the left posterior inferior frontal cortex is involved in binding syntactic frames together, whereas the left superior temporal cortex is involved in retrieval of the syntactic frames stored in memory. Lesion data that support the involvement of this left frontotemporal network in syntactic processing are discussed.

On performability: Structure and process in language understanding

This paper presents a process model of sentence comprehension. Drawing upon a modular conception of the language apparatus, the model distinguishes several levels of structural representation as well as several special-purpose processing mechanisms, including the verbal working memory system. The perspective gained from considering the functional architecture of language processing puts us in a position to advance specific hypotheses about the causes of impaired performance in populations with notable language difficulties. To test among these hypotheses, a methodological prescription is given for disentangling the subcomponents of the language apparatus which are intertwined in ordinary language use. This prescription is followed in an attempt to uncover the cognitive and linguistic processes implicated in reading disorder. We then proceed to consider empirical findings on both normal children and children who are reading disabled. The results lend support to the view that comprehension failures by poor ...