Distributional Semantic Models Research Articles

Determining the optimal environmental information for training computational models of lexical semantics and lexical organization.

Experiential theories of cognition propose that the external environment shapes cognitive processing, shifting emphasis from internal mechanisms to the learning of environmental structure. Computational modelling, particularly distributional models of lexical semantics (e.g., Landauer & Dumais, 1997) and models of lexical organization (e.g., Johns, 2021a), exemplifies this, highlights the influence of language experience on cognitive representations. While these models have been successful, comparatively less attention has been paid to the training materials used to train these models. Recent research has explored the role of social/communicatively oriented training materials on models of lexical semantics and organization (Johns, 2021a, 2021b, 2023, 2024), introducing discourse- and user-centred text training materials. However, determining the optimal training materials for these two model types remains an open question. This article addresses this problem by using experiential optimization (Johns, Jones, & Mewhort, 2019), which selects the materials that maximize model performance. This study will use experiential optimization to compare user-based and discourse-based corpora in optimizing models of lexical organization and semantics, offering insight into pathways towards integrating cognitive models in these areas. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Decomposing geographical judgments into spatial, temporal and linguistic components

When mentally exploring maps representing large-scale environments (e.g., countries or continents), humans are assumed to mainly rely on spatial information derived from direct perceptual experience (e.g., prior visual experience with the geographical map itself). In the present study, we rather tested whether also temporal and linguistic information could account for the way humans explore and ultimately represent this type of maps. We quantified temporal distance as the minimum time needed to travel by train across Italian cities, while linguistic distance was retrieved from natural language through cognitively plausible AI models based on non-spatial associative learning mechanisms (i.e., distributional semantic models). In a first experiment, we show that temporal and linguistic distances capture with high-confidence real geographical distances. Next, in a second behavioral experiment, we show that linguistic information can account for human performance over and above real spatial information (which plays the major role in explaining participants’ performance) in a task in which participants have to judge the distance between cities (while temporal information was found to be not relevant). These findings indicate that, when exploring maps representing large-scale environments, humans do take advantage of both perceptual and linguistic information, suggesting in turn that the formation of cognitive maps possibly relies on a strict interplay between spatial and non-spatial learning principles.

Exploring popular conceptions of democracy through media discourse: analysing dimensions of democracy from online media data in 93 countries using a distributional semantic model

ABSTRACT Survey studies show that popular support for democracy is strong in democratic and non-democratic countries. Naturally, the question is if democracy actually means the same thing in different linguistic, cultural, and political contexts. Mass media is often mentioned as decisive in forming citizens' understandings of democracy, but the media discourse is rarely in focus in comparative studies on popular conceptions of democracy. This article contributes to the debate by analysing data collected from online media in 93 countries. By utilizing tools from natural language processing, we provide new insights based on methods that are both extensive, flexible and cost-efficient. Our analysis shows that the media discourse revolves around democracy as governance, as outcomes and as values, but that these abstract understandings have additional dimensions. Our main contributions are three: (i) we show that the media discourse is related to popular understandings of democracy; (ii) our results indicate that there are common denominators of how the D-word is discussed in media across the globe, but when analysing the dimensions in more detail, common denominators are few and (iii) by relating democracy to everyday politics, media seems to legitimize any regime as democratic rather than being a beacon for liberal democracy.

Raising the Roof: Situating Verbs in Symbolic and Embodied Language Processing.

Recent investigations on how people derive meaning from language have focused on task-dependent shifts between two cognitive systems. The symbolic (amodal) system represents meaning as the statistical relationships between words. The embodied (modal) system represents meaning through neurocognitive simulation of perceptual or sensorimotor systems associated with a word's referent. A primary finding of literature in this field is that the embodied system is only dominant when a task necessitates it, but in certain paradigms, this has only been demonstrated using nouns and adjectives. The purpose of this paper is to study whether similar effects hold with verbs. Experiment 1 evaluated a novel task in which participants rated a selection of verbs on their implied vertical movement. Ratings correlated well with distributional semantic models, establishing convergent validity, though some variance was unexplained by language statistics alone. Experiment 2 replicated previous noun-based location-cue congruency experimental paradigms with verbs and showed that the ratings obtained in Experiment 1 predicted reaction times more strongly than language statistics. Experiment 3 modified the location-cue paradigm by adding movement to create an animated, temporally decoupled, movement-verb judgment task designed to examine the relative influence of symbolic and embodied processing for verbs. Results were generally consistent with linguistic shortcut hypotheses of symbolic-embodied integrated language processing; location-cue congruence elicited processing facilitation in some conditions, and perceptual information accounted for reaction times and accuracy better than language statistics alone. These studies demonstrate novel ways in which embodied and linguistic information can be examined while using verbs as stimuli.

Corpus linguistics meets historical linguistics and construction grammar: how far have we come, and where do we go from here?

This paper aims to give an overview of corpus-based research that investigates processes of language change from the theoretical perspective of Construction Grammar. Starting in the early 2000s, a dynamic community of researchers has come together in order to contribute to this effort. Among the different lines of work that have characterized this enterprise, this paper discusses the respective roles of qualitative approaches, diachronic collostructional analysis, multivariate techniques, distributional semantic models, and analyses of network structure. The paper tries to contextualize these approaches and to offer pointers for future research.

Probing the Representational Structure of Regular Polysemy via Sense Analogy Questions: Insights from Contextual Word Vectors.

Regular polysemes are sets of ambiguous words that all share the same relationship between their meanings, such as CHICKEN and LOBSTER both referring to an animal or its meat. To probe how a distributional semantic model, here exemplified by bidirectional encoder representations from transformers (BERT), represents regular polysemy, we analyzed whether its embeddings support answering sense analogy questions similar to "is the mapping between CHICKEN (as an animal) and CHICKEN (as a meat) similar to that which maps between LOBSTER (as an animal) to LOBSTER (as a meat)?" We did so using the LRcos model, which combines a logistic regression classifier of different categories (e.g., animal vs. meat) with a measure of cosine similarity. We found that (a) the model was sensitive to the shared structure within a given regular relationship; (b) the shared structure varies across different regular relationships (e.g., animal/meat vs. location/organization), potentially reflective of a "regularity continuum;" (c) some high-order latent structure is shared across different regular relationships, suggestive of a similar latent structure across different types of relationships; and (d) there is a lack of evidence for the aforementioned effects being explained by meaning overlap. Lastly, we found that both components of the LRcos model made important contributions to accurate responding and that a variation of this method could yield an accuracy boost of 10% in answering sense analogy questions. These findings enrich previous theoretical work on regular polysemy with a computationally explicit theory and methods, and provide evidence for an important organizational principle for the mental lexicon and the broader conceptual knowledgesystem.

Text Summarization for Online and Blended Learning

Online learning text summarization is vital for managing the constant influx of online information. It involves condensing lengthy online content into concise summaries while retaining the original meaning and information. While several online summarization tools are available, they often fall short in preserving the underlying semantics of the text. In this paper, we introduce an innovative approach to online text summarization that strongly emphasizes capturing and preserving the semantics of the text. Our automatic summarizer leverages distributional semantic models to extract and incorporate semantics, producing high-quality online summaries. To evaluate the effectiveness of our online summarization system, we conducted experiments on a diverse range of online content. We employed ROUGE metrics, a popular evaluation method for text summarization, to assess our system's performance. Additionally, we compared our results with those of four state-of-the-art online summarizers. The outcome of our study demonstrates that our online summarization approach, which integrates semantics as a fundamental feature, outperforms other reference summarizers. This conclusion underscores the significance of leveraging semantics in the context of online learning text summarization. Furthermore, our system's ability to reduce redundancies in online content makes it a valuable tool for managing information overload in the digital age.

Determining the Relativity of Word Meanings Through the Construction of Individualized Models of Semantic Memory.

Distributional models of lexical semantics are capable of acquiring sophisticated representations of word meanings. The main theoretical insight provided by these models is that they demonstrate the systematic connection between the knowledge that people acquire and the experience that they have with the natural language environment. However, linguistic experience is inherently variable and differs radically across people due to demographic and cultural variables. Recently, distributional models have been used to examine how word meanings vary across languages and it was found that there is considerable variability in the meanings of words across languages for most semantic categories. The goal of this article is to examine how variable word meanings are across individual language users within a single language. This was accomplished by assembling 500 individual user corpora attained from the online forum Reddit. Each user corpus ranged between 3.8 and 32.3 million words each, and a count-based distributional framework was used to extract word meanings for each user. These representations were then used to estimate the semantic alignment of word meanings across individual language users. It was found that there are significant levels of relativity in word meanings across individuals, and these differences are partially explained by other psycholinguistic factors, such as concreteness, semantic diversity, and social aspects of language usage. These results point to word meanings being fundamentally relative and contextually fluid, with this relativeness being related to the individualized nature of linguistic experience.

AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek

Abstract The last years have seen the application of Natural Language Processing, in particular, language models, to the study of the Semantics of ancient Greek, but only a little work has been done to create gold data for the evaluation of such models. In this contribution we introduce AGREE, the first benchmark for intrinsic evaluation of semantic models of ancient Greek created from expert judgements. In the absence of native speakers, eliciting expert judgements to create a gold standard is a way to leverage a competence that is the closest to that of natives. Moreover, this method allows for collecting data in a uniform way and giving precise instructions to participants. Human judgements about word relatedness were collected via two questionnaires: in the first, experts provided related lemmas to some proposed seeds, while in the second, they assigned relatedness judgements to pairs of lemmas. AGREE was built from a selection of the collected data.

Automatic Scoring of Verbal Divergent Thinking Tests: From Lexical Databases to Large Language Models

The article explores the evolution of methods for automatically assessing verbal divergent thinking tests. Resear - chers have increasingly focused on the ability to evaluate the originality of respondents' answers by calculating their semantic distance from the stimulus task. From 2009 to 2019, latent semantic analysis became the primary method for assessing semantic distances. Overall, in terms of internal consistency and correlation with expert ratings, its application yielded satisfactory results, maintaining an acceptable balance of quality and effort expended. However, issues emerged (dependence on the corpus used, result instability, systematic distortions related to the length of analyzed responses), prompting researchers to transition to more advanced models of distributional semantics (GloVe, Word2Vec etc.), large language models, and supervised learning. Large language models, especially those fine-tuned on creativity test materials, demonstrated higher effectiveness compared to models assessing semantic distances and approached expert evaluations. In addition to evaluating originality, the article considers works proposing methods for automatic assessment of elaboration, flexibility, associative flow, and divergent semantic integration. References to online platforms that allow for automatic assessments of originality in responses to divergent tests are provided. The issue of interpreting results obtained through large language models is discussed. A drawback of using these models is the lack of understanding of the basis on which judgments of the originality of creative products are made. The perspectives of applying explainable artificial intelligence for evaluating results of verbal and non-verbal tests of creative thinking are being discussed.

Lexical choice and word formation in a taboo game paradigm

We investigate the onomasiological question of which words speakers actually use and produce when trying to convey an intended meaning. This is not limited to selecting the best-fitting available existing word, but also includes word formation, the coinage of novel words. In the first two experiments, we introduce the taboo game paradigm in which participants were instructed to produce a single-word substitution for different words so that others can later identify them. Using distributional semantic models with the capability to produce quantitative representations for existing and novel word responses, we find that (a) responses tend to be semantically close to the targets and (b) existing words were represented closer than novel words, but (c) even novel compounds were often closer than the targets’ free associates. In a final third experiment, we find that other participants are more likely to guess the correct original word (a) for responses closer to the original targets, and (b) for novel compound responses as compared to existing word responses. This shows that the production of both existing and novel words can be accurately captured in a unified computational framework of the semantic mechanisms driving word choice.

AGRONER: An unsupervised agriculture named entity recognition using weighted distributional semantic model

AGRONER: An unsupervised agriculture named entity recognition using weighted distributional semantic model

Harnessing distributional semantics to build context-aware justifications for recommender systems

This paper introduces a methodology to generate review-based natural language justifications supporting personalized suggestions returned by a recommender system. The hallmark of our strategy lies in the fact that natural language justifications are adapted to the different contextual situations in which the items will be consumed. In particular, our strategy relies on the following intuition: Just like the selection of the most suitable item is influenced by the contexts of usage, a justification that supports a recommendation should vary as well. As an example, depending on whether a person is going out with her friends or her family, a justification that supports a restaurant recommendation should include different concepts and aspects. Accordingly, we designed a pipeline based on distributional semantics models to generate a vector space representation of each context. Such a representation, which relies on a term-context matrix, is used to identify the most suitable review excerpts that discuss aspects that are particularly relevant for a certain context. The methodology was validated by means of two user studies, carried out in two different domains (i.e., movies and restaurants). Moreover, we also analyzed whether and how our justifications impact on the perceived transparency of the recommendation process and allow the user to make more informed choices. As shown by the results, our intuitions were supported by the user studies.

Items Outperform Adjectives in a Computational Model of Binary Semantic Classification.

Semantic memory encompasses one's knowledge about the world. Distributional semantic models, which construct vector spaces with embedded words, are a proposed framework for understanding the representational structure of human semantic knowledge. Unlike some classic semantic models, distributional semantic models lack a mechanism for specifying the properties of concepts, which raises questions regarding their utility for a general theory of semantic knowledge. Here, we develop a computational model of a binary semantic classification task, in which participants judged target words for the referent's size or animacy. We created a family of models, evaluating multiple distributional semantic models, and mechanisms for performing the classification. The most successful model constructed two composite representations for each extreme of the decision axis (e.g., one averaging together representations of characteristically big things and another of characteristically small things). Next, the target item was compared to each composite representation, allowing the model to classify more than 1,500 words with human-range performance and to predict response times. We propose that when making a decision on a binary semantic classification task, humans use task prompts to retrieve instances representative of the extremes on that semantic dimension and compare the probe to those instances. This proposal is consistent with the principles of the instance theory of semantic memory.

Automated Assessment of Initial Answers to Questions in Conversational Intelligent Tutoring Systems: Are Contextual Embedding Models Really Better?

This paper assesses the ability of semantic text models to assess student responses to electronics questions compared with that of expert human judges. Recent interest in text similarity has led to a proliferation of models that can potentially be used for assessing student responses. However, it is unclear whether these models perform as well as early models of distributional semantics. We assessed 5166 response pairings of 219 participants across 118 electronics questions and scored each with 13 different computational text models, including models that use Regular Expressions, distributional semantics, embeddings, contextual embeddings, and combinations of these features. Regular Expressions performed the best out of the stand-alone models. We show other semantic text models performing comparably to the Latent Semantic Analysis model that was originally used for the current task, and in a small number of cases outperforming the model. Models trained on a domain-specific electronics corpus for the task performed better than models trained on general language or Newtonian physics. Furthermore, semantic text models combined with RegEx outperformed stand-alone models in agreement with human judges. Tuning the performance of these recent models in Automatic Short Answer Grading tasks for conversational intelligent tutoring systems requires empirical analysis, especially in domain-specific areas such as electronics. Therefore, the question arises as to how well recent contextual embedding models compare with earlier distributional semantic language models on this task of answering questions about electronics. These results shed light on the selection of appropriate computational techniques for text modeling to improve the accuracy, recall, weighted agreement, and ultimately the effectiveness of automatic scoring in conversational ITSs.

A common neural code for meaning in discourse production and comprehension

How does the brain code the meanings conveyed by language? Neuroimaging studies have investigated this by linking neural activity patterns during discourse comprehension to semantic models of language content. Here, we applied this approach to the production of discourse for the first time. Participants underwent fMRI while producing and listening to discourse on a range of topics. We used a distributional semantic model to quantify the similarity between different speech passages and identified where similarity in neural activity was predicted by semantic similarity. When people produced discourse, speech on similar topics elicited similar activation patterns in a widely distributed and bilateral brain network. This network was overlapping with, but more extensive than, the regions that showed similarity effects during comprehension. Critically, cross-task neural similarities between comprehension and production were also predicted by similarities in semantic content. This result suggests that discourse semantics engages a common neural code that is shared between comprehension and production. Effects of semantic similarity were bilateral in all three RSA analyses, even while univariate activation contrasts in the same data indicated left-lateralised BOLD responses. This indicates that right-hemisphere regions encode semantic properties even when they are not activated above baseline. We suggest that right-hemisphere regions play a supporting role in processing the meaning of discourse during both comprehension and production.

ViSpa (Vision Spaces): A computer-vision-based representation system for individual images and concept prototypes, with large-scale evaluation.

Quantitative, data-driven models for mental representations have long enjoyed popularity and success in psychology (e.g., distributional semantic models in the language domain), but have largely been missing for the visual domain. To overcome this, we present ViSpa (Vision Spaces), high-dimensional vector spaces that include vision-based representation for naturalistic images as well as concept prototypes. These vectors are derived directly from visual stimuli through a deep convolutional neural network trained to classify images and allow us to compute vision-based similarity scores between any pair of images and/or concept prototypes. We successfully evaluate these similarities against human behavioral data in a series of large-scale studies, including off-line judgments-visual similarity judgments for the referents of word pairs (Study 1) and for image pairs (Study 2), and typicality judgments for images given a label (Study 3)-as well as online processing times and error rates in a discrimination (Study 4) and priming task (Study 5) with naturalistic image material. ViSpa similarities predict behavioral data across all tasks, which renders ViSpa a theoretically appealing model for vision-based representations and a valuable research tool for data analysis and the construction of experimental material: ViSpa allows for precise control over experimental material consisting of images and/or words denoting imageable concepts and introduces a specifically vision-based similarity for word pairs. To make ViSpa available to a wide audience, this article (a) includes (video) tutorials on how to use ViSpa in R and (b) presents a user-friendly web interface at http://vispa.fritzguenther.de. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Exploring terminological relations between multi-word terms in distributional semantic models

Abstract A term is a lexical unit with specialized meaning in a particular domain. Terms may be simple (STs) or multi-word (MWTs). The organization of terms gives a representation of the structure of domain knowledge, which is based on the relationships between the concepts of the domain. However, relations between MWTs are often underrepresented in terminology resources. This work aims to explore distributional semantic models for capturing terminological relations between multi-word terms through lexical substitution and analogy. The experiments show that the results of the analogy-based method are globally better than those of the one based on lexical substitution and that analogy is well suited to the acquisition of synonymy, antonymy, and hyponymy while lexical substitution performs best for hypernymy.

Investigating the Extent to which Distributional Semantic Models Capture a Broad Range of Semantic Relations.

Distributional semantic models (DSMs) are a primary method for distilling semantic information from corpora. However, a key question remains: What types of semantic relations among words do DSMs detect? Prior work typically has addressed this question using limited human data that are restricted to semantic similarity and/or general semantic relatedness. We tested eight DSMs that are popular in current cognitive and psycholinguistic research (positive pointwise mutual information; global vectors; and three variations each of Skip-gram and continuous bag of words (CBOW) using word, context, and mean embeddings) on a theoretically motivated, rich set of semantic relations involving words from multiple syntactic classes and spanning the abstract-concrete continuum (19 sets of ratings). We found that, overall, the DSMs are best at capturing overall semantic similarity and also can capture verb-noun thematic role relations and noun-noun event-based relations that play important roles in sentence comprehension. Interestingly, Skip-gram and CBOW performed the best in terms of capturing similarity, whereas GloVe dominated the thematic role and event-based relations. We discuss the theoretical and practical implications of our results, make recommendations for users of these models, and demonstrate significant differences in model performance on event-based relations.

A study on surprisal and semantic relatedness for eye-tracking data prediction.

Previous research in computational linguistics dedicated a lot of effort to using language modeling and/or distributional semantic models to predict metrics extracted from eye-tracking data. However, it is not clear whether the two components have a distinct contribution, with recent studies claiming that surprisal scores estimated with large-scale, deep learning-based language models subsume the semantic relatedness component. In our study, we propose a regression experiment for estimating different eye-tracking metrics on two English corpora, contrasting the quality of the predictions with and without the surprisal and the relatedness components. Different types of relatedness scores derived from both static and contextual models have also been tested. Our results suggest that both components play a role in the prediction, with semantic relatedness surprisingly contributing also to the prediction of function words. Moreover, they show that when the metric is computed with the contextual embeddings of the BERT model, it is able to explain a higher amount of variance.