Neural Word Representations Research Articles

Atypical hemispheric re-organization of the reading network in high-functioning adults with dyslexia: Evidence from representational similarity analysis

Abstract It has been argued that university students with dyslexia compensate for their reading deficits by a neural re-organization of the typical reading network, where the lexical representations of words are (re-)structured according to semantic rather than orthographic information. To investigate the re-organization of neural word representations more directly, we used multivariate representational similarity analyses (RSA) to find out which brain regions of the reading network respond to orthographic and semantic similarity between 544 pairs of words and whether there were any differences between typical and dyslexic readers. In accordance with the re-organization hypothesis, we predicted greater similarity (i.e., correlation of neural dissimilarity matrices) in adult dyslexic than in typical readers in regions associated with semantic processing and weaker similarity in regions associated with orthographic processing. Our results did not confirm these predictions. First, we found sensitivity to semantic similarity in all three subparts of the fusiform gyrus (FG1, FG2, and FG3) bilaterally. Adults with dyslexia showed less (rather than more) sensitivity to semantic similarity in the posterior subpart of fusiform gyrus (FG1) in the left hemisphere. Second, in typical readers, sensitivity to orthographic information was not only found in the left fusiform gyrus (FG1, FG2, and FG3) but also in left inferior frontal gyrus (IFG). Adults with dyslexia, in contrast, did not show sensitivity to orthographic information in left IFG. However, they showed increased sensitivity to orthographic information in the right hemisphere FG1. Together, the results show abnormal orthographic processing in left IFG and right FG1 and reduced semantic information in left FG1. While we found evidence for compensatory re-organization in adult dyslexia, the present results do not support the hypothesis according to which adults with dyslexia rely more heavily on semantic information. Instead, they revealed atypical hemispheric organization of the reading network that is not restricted to the typical left language hemisphere.

Delta-Band Neural Responses to Individual Words Are Modulated by Sentence Processing.

To understand language, we need to recognize words and combine them into phrases and sentences. During this process, responses to the words themselves are changed. In a step toward understanding how the brain builds sentence structure, the present study concerns the neural readout of this adaptation. We ask whether low-frequency neural readouts associated with words change as a function of being in a sentence. To this end, we analyzed an MEG dataset by Schoffelen et al. (2019) of 102 human participants (51 women) listening to sentences and word lists, the latter lacking any syntactic structure and combinatorial meaning. Using temporal response functions and a cumulative model-fitting approach, we disentangled delta- and theta-band responses to lexical information (word frequency), from responses to sensory and distributional variables. The results suggest that delta-band responses to words are affected by sentence context in time and space, over and above entropy and surprisal. In both conditions, the word frequency response spanned left temporal and posterior frontal areas; however, the response appeared later in word lists than in sentences. In addition, sentence context determined whether inferior frontal areas were responsive to lexical information. In the theta band, the amplitude was larger in the word list condition ∼100 milliseconds in right frontal areas. We conclude that low-frequency responses to words are changed by sentential context. The results of this study show how the neural representation of words is affected by structural context and as such provide insight into how the brain instantiates compositionality in language.SIGNIFICANCE STATEMENT Human language is unprecedented in its combinatorial capacity: we are capable of producing and understanding sentences we have never heard before. Although the mechanisms underlying this capacity have been described in formal linguistics and cognitive science, how they are implemented in the brain remains to a large extent unknown. A large body of earlier work from the cognitive neuroscientific literature implies a role for delta-band neural activity in the representation of linguistic structure and meaning. In this work, we combine these insights and techniques with findings from psycholinguistics to show that meaning is more than the sum of its parts; the delta-band MEG signal differentially reflects lexical information inside and outside sentence structures.

Deep analysis of word sense disambiguation via semi-supervised learning and neural word representations

• Ambiguity is a challenging task in text mining addressed for word-sense disambiguation algorithms. • The lack of labeled dataset is a barrier and semi-supervised learning (SSL) addresses this problem. • Our method explored SSL and word-embeddings CBOW, SKIP-GRAM, FASTTEXT, GLOVE, BERT and ELECTRA . • The F1-score increases in several datasets like Senseval-2, Senseval-3, Semeval-2007 and Semcor. Word Sense Disambiguation (WSD) aims to determine the meaning of a word in context. Different approaches have been proposed in supervised and unsupervised domains. In most cases, supervised learning provides superior WSD performance. Since sense-annotated corpora can be difficult or time-consuming to obtain, which must be repeated for new domains, languages, and sense inventories, semi-supervised learning (SSL) methods, that combine a small amount of sense-annotated data, start to be pre-eminent. In SSL, graph-based methods are common, because they capture the relationships between terms using an undirected graph. This paper aims to investigate semi-supervised WSD by considering different graph-based SSL algorithms with features generated by word embeddings from Word2Vec, FastText, GloVe, BERT and ELECTRA models combined with parts-of-speech tags and word context. We test several combinations of word-embedding models, similarity measures for graph construction and SSL classification algorithms to disambiguate classical lexical sample WSD datasets. The results indicate our SSL algorithms achieved competitive results compared to supervised ones and the ELECTRA models performed better than other embeddings for SSL.

Neural representation of words within phrases: Temporal evolution of color-adjectives and object-nouns during simple composition.

In language, stored semantic representations of lexical items combine into an infinitude of complex expressions. While the neuroscience of composition has begun to mature, we do not yet understand how the stored representations evolve and morph during composition. New decoding techniques allow us to crack open this very hard question: we can train a model to recognize a representation in one context or time-point and assess its accuracy in another. We combined the decoding approach with magnetoencephalography recorded during a picture naming task to investigate the temporal evolution of noun and adjective representations during speech planning. We tracked semantic representations as they combined into simple two-word phrases, using single words and two-word lists as non-combinatory controls. We found that nouns were generally more decodable than adjectives, suggesting that noun representations were stronger and/or more consistent across trials than those of adjectives. When training and testing across contexts and times, the representations of isolated nouns were recoverable when those nouns were embedded in phrases, but not so if they were embedded in lists. Adjective representations did not show a similar consistency across isolated and phrasal contexts. Noun representations in phrases also sustained over time in a way that was not observed for any other pairing of word class and context. These findings offer a new window into the temporal evolution and context sensitivity of word representations during composition, revealing a clear asymmetry between adjectives and nouns. The impact of phrasal contexts on the decodability of nouns may be due to the nouns' status as head of phrase-an intriguing hypothesis for future research.

Neural text similarity of user reviews for improving collaborative filtering recommender systems

Abstract According to the advent of technology and the expansion of using the World Wide Web, there has been an enormous increase in the number of Internet retailers and their customers. The amount of data is exploding due to the numerous users and the variety of products. In such a condition, recommender systems play an important role in helping users and providing suggestions based on their taste. Using recommender systems’ suggestions, users can save their time on finding their personalized, useful, and favorite items without being overwhelmed by a large set of items. User-based collaborative filtering recommender systems for each user find similar users based on their ratings and suggest their favorite products to the intended user. In this paper, we present a model to improve recommender systems by finding similar users based on their reviews in addition to their ratings. To this end, we compute users’ reviews similarity utilizing seven different approaches, out of which two techniques are lexical-based, two techniques benefit from the neural representation of words, and three techniques are based on the neural representation of texts. Our experiments on two different datasets from Amazon show that the proposed model for capturing review’s similarity significantly improved the performance of the recommender system. Moreover, the model based on Long Short Term Memory (LSTM) network achieves the best results.

Information Extraction from Text Intensive and Visually Rich Banking Documents

Document types, where visual and textual information plays an important role in their analysis and understanding, pose a new and attractive area for information extraction research. Although cheques, invoices, and receipts have been studied in some previous multi-modal studies, banking documents present an unexplored area due to the naturalness of the text they possess in addition to their visual richness. This article presents the first study which uses visual and textual information for deep-learning based information extraction on text-intensive and visually rich scanned documents which are, in this instance, unstructured banking documents, or more precisely, money transfer orders. The impact of using different neural word representations (i.e., FastText, ELMo, and BERT) on IE subtasks (namely, named entity recognition and relation extraction stages), positional features of words on document images and auxiliary learning with some other tasks are investigated. The article proposes a new relation extraction algorithm based on graph factorization to solve the complex relation extraction problem where the relations within documents are n-ary, nested, document-level, and previously indeterminate in quantity. Our experiments revealed that the use of deep learning algorithms yielded around 10 percentage points improvement on the IE sub-tasks. The inclusion of word positional features yielded around 3 percentage points of improvement in some specific information fields. Similarly, our auxiliary learning experiments yielded around 2 percentage points of improvement on some information fields associated with the specific transaction type detected by our auxiliary task. The integration of the information extraction system into a real banking environment reduced cycle times substantially. When compared to the manual workflow, document processing pipeline shortened book-to-book money transfers to 10 minutes (from 29 min.) and electronic fund transfers (EFT) to 17 minutes (from 41 min.) respectively.

An Analysis of Neural Word Representations for Wikipedia Articles Classification

One of the current popular methods of generating word representations is an approach based on the analysis of large document collections with neural networks. It creates so-called word-embeddings that attempt to learn relationships between words and encode this information in the form of a low-dimensional vector. The goal of this paper is to examine the differences between the most popular embedding models and the typical bag-of-words (BoW) approach used for document representation. The hypothesis behind the experiments is that the more informative the representation is, the better classification results it produces. The evaluation of the representations has been performed with regards to the accuracy of three text classifiers. The experiments have been performed on subsets of articles selected from Wikipedia. To test the independence of the results from the language of their dataset, we created datasets for the Polish and English versions of this repository. The datasets have been provided publicly to create a baseline to study the different representation methods. The classification tasks, which aim to reconstruct the human-made Wikipedia categories, confirm that the word embeddings can be successfully used for text classification. Word embeddings for document representation with typical vector averaging methods does not outperform usage of BoW. We use a modification of the document representation based on kernel transformations that shows an improvement of the text classification results. Also, we find that in most cases the method of dimensionality reduction with neural embeddings outperforms that of LSA.

Orthographic and Phonological Representations in the Fusiform Cortex.

Mental and neural representations of words are at the core of understanding the cognitive and neural mechanisms of reading. Despite extensive studies, the nature of visual word representation remains highly controversial due to methodological limitations. In particular, it is unclear whether the fusiform cortex contains only abstract orthographic representation, or represents both lower and higher level orthography as well as phonology. Using representational similarity analysis, we integrated behavioral ratings, computational models of reading and visual object recognition, and neuroimaging data to examine the nature of visual word representations in the fusiform cortex. Our results provided clear evidence that the middle and anterior fusiform represented both phonological and orthographic information. Whereas lower level orthographic information was represented at every stage of the ventral visual stream, abstract orthographic information was increasingly represented along the posterior-to-anterior axis. Furthermore, the left and right hemispheres were tuned to high- and low-frequency orthographic information, respectively. These results help to resolve the long-standing debates regarding the role of the fusiform in reading, and have significant implications for the development of psychological, neural, and computational theories of reading.

The influence of lexical statistics on temporal lobe cortical dynamics during spoken word listening

Neural representations of words are thought to have a complex spatio-temporal cortical basis. It has been suggested that spoken word recognition is not a process of feed-forward computations from phonetic to lexical forms, but rather involves the online integration of bottom-up input with stored lexical knowledge. Using direct neural recordings from the temporal lobe, we examined cortical responses to words and pseudowords. We found that neural populations were not only sensitive to lexical status (real vs. pseudo), but also to cohort size (number of words matching the phonetic input at each time point) and cohort frequency (lexical frequency of those words). These lexical variables modulated neural activity from the posterior to anterior temporal lobe, and also dynamically as the stimuli unfolded on a millisecond time scale. Our findings indicate that word recognition is not purely modular, but relies on rapid and online integration of multiple sources of lexical knowledge.

The visual representations of words and style in text: An adaptation study

While the nature of face representations in the human perceptual system has been extensively studied using adaptation, there has been little investigation using this technique of the neural basis of another parallel class of high-level objects, words. We used the perceptual-bias technique to determine if aftereffects could be generated for either the word content or stylistic properties of textual stimuli, and if these aftereffects showed invariance for the non-adapted dimension. In a first experiment, we examined adaptation for word versus handwriting style. In a second experiment we contrasted adaptation for words with adaptation for computer font. The third experiment performed a similar study of aftereffects for words and case. In all three experiments we consistently found adaptation for words, which were not diminished by changing the style between the adapting and probe stimuli: hence word aftereffects are invariant for handwriting, font and case. Aftereffects were negligible for style. Additional analyses showed that discriminative ability was better for word than for style content. These results confirm that the neural representations of words can be probed with the adaptation technique and suggest that adaptation accesses word representations at an abstract level, where the identity of a word is invariant for stylistic properties.

Are abstract action words embodied? An fMRI investigation at the interface between language and motor cognition

The cognitive and neural representation of abstract words is still an open question for theories of embodied cognition. Generally, it is proposed that abstract words are grounded in the activation of sensorimotor or at least experiential properties, exactly as concrete words. Further behavioral theories propose multiple representations evoked by abstract and concrete words. We conducted a functional magnetic resonance imaging (fMRI) study to investigate the neural correlates of concrete and abstract multi-word expressions in an action context. Participants were required to read simple sentences which combined each concrete noun with an adequate concrete verb and an adequate abstract verb, as well as an adequate abstract noun with either kind of verbs previously used. Thus, our experimental design included a continuum from pure concreteness to mere abstractness. As expected, comprehension of both concrete and abstract language content activated the core areas of the sensorimotor neural network namely the left lateral (precentral gyrus) and medial (supplementary motor area) premotor cortex. While the purely concrete multi-word expressions elicited activations within the left inferior frontal gyrus (pars triangularis) and two foci within the left inferior parietal cortex, the purely abstract multi-word expressions were represented in the anterior part of left middle temporal gyrus that is part of the language processing system. Although the sensorimotor neural network is engaged in both concrete and abstract language contents, the present findings show that concrete multi-word processing relies more on the sensorimotor system, and abstract multi-word processing relies more on the linguistic system.

Gestures Enhance Foreign Language Learning

Language and gesture are highly interdependent systems that reciprocally influence each other. For example, performing a gesture when learning a word or a phrase enhances its retrieval compared to pure verbal learning. Although the enhancing effects of co-speech gestures on memory are known to be robust, the underlying neural mechanisms are still unclear. Here, we summarize the results of behavioral and neuroscientific studies. They indicate that the neural representation of words consists of complex multimodal networks connecting perception and motor acts that occur during learning. In this context, gestures can reinforce the sensorimotor representation of a word or a phrase, making it resistant to decay. Also, gestures can favor embodiment of abstract words by creating it from scratch. Thus, we propose the use of gesture as a facilitating educational tool that integrates body and mind.

Commonality of neural representations of words and pictures

In this work we explore whether the patterns of brain activity associated with thinking about concrete objects are dependent on stimulus presentation format, whether an object is referred to by a written or pictorial form. Multi-voxel pattern analysis methods were applied to brain imaging (fMRI) data to identify the item category associated with brief viewings of each of 10 words (naming 5 tools and 5 dwellings) and, separately, with brief viewings of each of 10 pictures (line drawings) of the objects named by the words. These methods were able to identify the category of the picture the participant was viewing, based on neural activation patterns observed during word-viewing, and identify the category of the word the participant was viewing, based on neural activation patterns observed during picture-viewing, using data from only that participant or only from other participants. These results provide an empirical demonstration of object category identification across stimulus formats and across participants. In addition, we were able to identify the category of the word that the participant was viewing based on the patterns of neural activation generated during word-viewing by that participant or by all other participants. Similarly, we were able to identify with even higher accuracy the category of the picture the participant was viewing, based on the patterns of neural activation demonstrated during picture-viewing by that participant or by all other participants. The brain locations that were important for category identification were similar across participants and were distributed throughout the cortex where various object properties might be neurally represented. These findings indicate consistent triggering of semantic representations using different stimulus formats and suggest the presence of stable, distributed, and identifiable neural states that are common to pictorial and verbal input referring to object categories.

The Role of Semantics and Grammatical Class in the Neural Representation of Words

On the basis of neuropsychological and functional imaging evidence, meaning and grammatical class (particularly the verb-noun distinction) have been proposed as organizational principles of linguistic knowledge in the brain. However, previous studies investigating verb and noun processing have been confounded by the presence of systematic correlations between word meaning and grammatical class. In this positron emission tomography study, we investigated implicit word processing using stimuli that allowed the effects of semantic and grammatical properties to be examined independently, without grammatical-semantic confounds. We found that left hemisphere cortical activation during single-word processing was modulated by word meaning, but not by grammatical class. Motor word processing produced significant activation in left precentral gyrus, whereas sensory word processing produced significant activation in left inferior temporal and inferior frontal regions. In contrast to previous studies, there were no effects of grammatical class in left inferior frontal gyrus (IFG). Instead, we found semantic-based differences within left IFG: anterior, but not posterior, left IFG regions responded preferentially to sensory words. These findings demonstrate that the neural substrates of implicit word processing are determined by semantic rather than grammatical properties and suggest that word comprehension involves the activation of modality-specific representations linked to word meaning.

Hemispheric specialization and independence for word recognition: A comparison of three computational models

Two findings serve as the hallmark for hemispheric specialization during lateralized lexical decision. First is an overall word advantage, with words being recognized more quickly and accurately than non-words (the effect being stronger in response latency). Second, a right visual field advantage is observed for words, with little or no hemispheric differences in the ability to identify non-words. Several theories have been proposed to account for this difference in word and non-word recognition, some by suggesting dual routes of lexical access and others by incorporating separate, and potentially independent, word and non-word detection mechanisms. We compare three previously proposed cognitive theories of hemispheric interactions (callosal relay, direct access, and cooperative hemispheres) through neural network modeling, with each network incorporating different means of interhemispheric communication. When parameters were varied to simulate left hemisphere specialization for lexical decision, only the cooperative hemispheres model showed both a consistent left hemisphere advantage for word recognition but not non-word recognition, as well as an overall word advantage. These results support the theory that neural representations of words are more strongly established in the left hemisphere through prior learning, despite open communication between the hemispheres during both learning and recall.