Word Co-occurrence Statistics Research Articles

Distributional hypothesis as isomorphism between word-word co-occurrence and analogical parallelograms.

Most of the modern natural language processing (NLP) techniques are based on the vector space models of language, in which each word is represented by a vector in a high dimensional space. One of the earliest successes was demonstrated by the four-term analogical reasoning task: what is to C as B is to A? The trained word vectors form "parallelograms" representing the quadruple of words in analogy. This discovery in NLP offers us insight into our understanding of human semantic representation of words via analogical reasoning. Despite successful applications of the large-scale language models, it has not been fully understood why such parallelograms emerge by learning through natural language data. As the vector space model is not optimized to form parallelograms, the key structure related to geometric shapes of word vectors is expected to be in the data, rather than the models. In the present article, we test our hypothesis that such parallelogram arrangement of word vectors readily exists in the co-occurrence statistics of language. Our approach focuses more on the data itself, and it is different from the existing theoretical approach trying to find the mechanism of parallelogram formation in the algorithms and/or vector arithmetic operations on word vectors. First, our analysis suggested that analogical reasoning is possible by decomposition of the bigram co-occurrence matrix. Second, we demonstrated the formation of a parallelepiped, a more structured geometric object than a parallelogram, by creating a small artificial corpus and its word vectors. With these results, we propose a refined form of distributional hypothesis pointing out an isomorphism between a sort of symmetry or exchangeability and word co-occurrence statistics.

Diachronic predictor of affective ratings of English words

This work considers implementation of a diachronic predictor of valence, arousal and dominance ratings of English words. The estimation of affective ratings is based on data on word co-occurrence statistics in the large diachronic Google Books Ngram corpus. Affective ratings from the NRC VAD dictionary are used as target values for training. When tested on synchronic data, the obtained Pearson‘s correlation coefficients between human affective ratings and their machine ratings are 0.843, 0.779 and 0.792 for valence, aroused and dominance, respectively. We also provide a detailed analysis of the accuracy of the predictor on diachronic data. The main result of the work is creation of a diachronic affective dictionary of English words. Several examples are considered that illustrate jumps in the time series of affective ratings when a word gains a new meaning. This indicates that changes in affective ratings can serve as markers of lexical-semantic changes.

The effects of speaker and exemplar variability in children's cross-situational word learning.

Cross-situational word learning (XSWL) - children's ability to learn words by tracking co-occurrence statistics of words and their referents over time - has been identified as a fundamental mechanism underlying lexical learning. However, it is unknown whether children can acquire new words when faced with variable input in XSWL paradigms, such as varying object exemplars and variable speakers. In the present study, we examine the separate and combined effects of exemplar and speaker variability on XSWL in typically developing English-speaking monolingual children. Results revealed that variability in speakers and exemplars did not facilitate or hinder XSWL performance. However, input that varied in both speakers and exemplars simultaneously did hinder children's word learning. Results from this work suggest that XSWL mechanisms may support categorization and generalization beyond word-object associations, but that accommodating multiple forms of variable input may incur costs. Overall, this research provides new theoretical insights into how fundamental mechanisms of word learning scale to more complex and naturalistic forms of input.

MarQO: A query optimizer in multilingual environment for information retrieval in Marathi language

Information retrieval is a crucial component of modern information systems. A significant portion of the vast amount of information stored worldwide is in local languages. While most information retrieval systems are designed primarily for English, there is a growing need for these systems to work with data in languages other than English. Cross Language Information Retrieval (CLIR) systems play a pivotal role in enabling information retrieval across multiple languages. However, these systems often face challenges due to ambiguities in query translation, impacting retrieval accuracy. This paper introduces "MarQO," a query optimizer designed to address these challenges in the context of Marathi language. MarQO employs a multi-stage approach, including lexical processing, extraction of multi-word terms, synonym addition, phrasal translations, utilization of word co-occurrence statistics, and more. By disambiguating query keyword translations, MarQO significantly improves the accuracy of translations, thereby leading to more relevant document retrieval results.

Synwmd: Syntax-aware word Mover’s distance for sentence similarity evaluation

Word Mover’s Distance (WMD) computes the distance between words and models text similarity with the moving cost between words in two text sequences. Yet, it does not offer good performance in sentence similarity evaluation since it does not incorporate word importance and fails to take inherent contextual and structural information in a sentence into account. An improved WMD method using the syntactic parse tree, called Syntax-aware Word Mover’s Distance (SynWMD), is proposed to address these two shortcomings in this work. First, a weighted graph is built upon the word co-occurrence statistics extracted from the syntactic parse trees of sentences. The importance of each word is inferred from graph connectivities. Second, the local syntactic parsing structure of words is considered in computing the distance between words. To demonstrate the effectiveness of the proposed SynWMD, we conduct experiments on 6 textual semantic similarity (STS) datasets and 4 sentence classification datasets. Experimental results show that SynWMD achieves state-of-the-art performance on STS tasks. It also outperforms other WMD-based methods on sentence classification tasks.

Synwmd: Syntax-Aware Word Mover's Distance for Sentence Similarity Evaluation

Word Mover's Distance (WMD) computes the distance between words and models text similarity with the moving cost between words in two text sequences. Yet, it does not offer good performance in sentence similarity evaluation since it does not incorporate word importance and fails to take inherent contextual and structural information in a sentence into account. An improved WMD method using the syntactic parse tree, called Syntax-aware Word Mover's Distance (SynWMD), is proposed to address these two shortcomings in this work. First, a weighted graph is built upon the word co-occurrence statistics extracted from the syntactic parse trees of sentences. The importance of each word is inferred from graph connectivities. Second, the local syntactic parsing structure of words is considered in computing the distance between words. To demonstrate the effectiveness of the proposed SynWMD, we conduct experiments on 6 textual semantic similarity (STS) datasets and 4 sentence classification datasets. Experimental results show that SynWMD achieves state-of-the-art performance on STS tasks. It also outperforms other WMD-based methods on sentence classification tasks.

Semantic Structure in Deep Learning

Deep learning has recently come to dominate computational linguistics, leading to claims of human-level performance in a range of language processing tasks. Like much previous computational work, deep learning–based linguistic representations adhere to the distributional meaning-in-use hypothesis, deriving semantic representations from word co-occurrence statistics. However, current deep learning methods entail fundamentally new models of lexical and compositional meaning that are ripe for theoretical analysis. Whereas traditional distributional semantics models take a bottom-up approach in which sentence meaning is characterized by explicit composition functions applied to word meanings, new approaches take a top-down approach in which sentence representations are treated as primary and representations of words and syntax are viewed as emergent. This article summarizes our current understanding of how well such representations capture lexical semantics, world knowledge, and composition. The goal is to foster increased collaboration on testing the implications of such representations as general-purpose models of semantics.

Distributional social semantics: Inferring word meanings from communication patterns

Distributional models of lexical semantics have proven to be powerful accounts of how word meanings are acquired from the natural language environment (Günther, Rinaldi, & Marelli, 2019; Kumar, 2020). Standard models of this type acquire the meaning of words through the learning of word co-occurrence statistics across large corpora. However, these models ignore social and communicative aspects of language processing, which is considered central to usage-based and adaptive theories of language (Tomasello, 2003; Beckner et al., 2009). Johns (2021) recently demonstrated that integrating social and communicative information into a lexical strength measure allowed for benchmark fits to be attained for lexical organization data, indicating that the social world contains important statistical information for language learning and processing. Through the analysis of the communication patterns of over 330,000 individuals on the online forum Reddit, totaling approximately 55 billion words of text, the findings of the current article demonstrates that social information about word usage allows for unique aspects of a word’s meaning to be acquired, providing a new pathway for distributional model development.

A novel co-occurrence-based approach to predict pure associative and semantic priming.

The theoretical "difficulty in separating association strength from [semantic] feature overlap" has resulted in inconsistent findings of either the presence or absence of "pure" associative priming in recent literature (Hutchison, 2003, Psychonomic Bulletin & Review, 10(4), p. 787). The present study used co-occurrence statistics of words in sentences to provide a full factorial manipulation of direct association (strong/no) and the number of common associates (many/no) of the prime and target words. These common associates were proposed to serve as semantic features for a recent interactive activation model of semantic processing (i.e., the associative read-out model; Hofmann & Jacobs, 2014). With stimulus onset asynchrony (SOA) as an additional factor, our findings indicate that associative and semantic priming are indeed dissociable. Moreover, the effect of direct association was strongest at a long SOA (1,000 ms), while many common associates facilitated lexical decisions primarily at a short SOA (200 ms). This response pattern is consistent with previous performance-based accounts and suggests that associative and semantic priming can be evoked by computationally determined direct and common associations.

워드임베딩과 그래프 기반 준지도학습을 통한 한국어 어휘 감성 점수 산출

Sentiment analysis plays an important role in both public and private sectors to understand consumers’ responses to products or voters’ reactions to policies. One of the most key success factors of sentiment analysis is to build an appropriate sentiment word dictionary. Many current existing approaches either heavily rely on the knowledge of domain experts or word co-occurrence statistics, the first of which causes low efficiency and high expenditure while the second of which suffers from incomplete data. In order to resolve these shortcomings, we propose a new domain-specific Korean word sentiment score evaluation method based on word embedding and graph based semi-supervised learning. First, words are embedded in a lower dimensional space by Word2Vec technique. Then, the word relation graph is constructed based on the similarity between words in the embedding space. Then, we assign sentiments to approximately 1% words utilizing some indicators like centrality measure. The sentiment scores of the other unlabeled words are automatically assigned by label propagation with semisupervised learning. To verify our proposed method, we collect 1.98 million review comments from three movie review websites. Experimental results show that the proposed method achieves about 93% accuracy of polarity classification.

Multimodal Word Meaning Induction From Minimal Exposure to Natural Text.

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models (DSMs) have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a set of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition.

Modeling Pauses for Synthesis of Storytelling Style Speech Using Unsupervised Word Features

Abstract In the storytelling style speech pauses or phrase breaks play a significant role in introducing suspense and climax in the story. More often pauses are provided by a storyteller to capture the audience's attention by emphasizing keywords, focusing emotion-salient words, and to separate key phrases in an utterance. The goal of the work presented in this paper is to predict the location of pauses, in an utterance synthesized by a Story Text-To-Speech (TTS) system using unsupervised features at word-level. Traditional methods for predicting pauses uses the foremost linguistic features like Parts-of-Speech (POS) tags, chunking information or terminal syllables, etc. These methods presuppose the availability of linguistic knowledge by an automatic tagger or manually annotated corpus. However, this information's are not readily available in case of Indian Languages. Manually annotating the text with this linguistic information is quite hectic and time consuming. Also, these pieces of information's do not capture the co-occurrence statistics of words. Hence, we propose a framework for integrating the Story TTS with proposed pause prediction module. In this module, an unlabeled text corpus is used to extract, the continuous-valued world-level features to model the pause patterns in storytelling speech. A set of story-specific (SS) features are introduced for capturing story-semantic information based on pause pattern. A various combination of pause predictions systems is- proposed such as B, POS, U, POS+SS and U+SS. These systems are evaluated objectively by F-1 Score.

Semantic variability predicts neural variability of object concepts

The prevailing approach to the neuroscientific study of concepts is to characterize the neural pattern evoked by a given concept, averaging over any variation that might occur upon multiple retrieval attempts (e.g., across time, tasks, or people). This approach—which diverges substantially from approaches to studying conceptual processing with other methods—treats all variation as noise. Here, our goal is to determine whether variation in neural patterns evoked by semantic retrieval of a given concept is more than just measurement error, and instead reflects variation arising from contextual variability. We measured each concept's diversity of semantic contexts (“SV”) by analyzing its word frequency and co-occurrence statistics in large text corpora. To measure neural variability, we conducted an fMRI study and sampled neural activity associated with each concept when it appeared in three separate, randomized contexts. We predicted that concepts with low SV would exhibit uniform activation patterns across stimulus presentations, whereas concepts with high SV would exhibit more dynamic representations over time. We observed that a concept's SV score predicted its corresponding neural variability. This finding supports a flexible, distributed organization of semantic memory, where a concept's meaning and its neural activity patterns both continuously vary across contexts.

Parsing Models for Identifying Multiword Expressions

Multiword expressions lie at the syntax/semantics interface and have motivated alternative theories of syntax like Construction Grammar. Until now, however, syntactic analysis and multiword expression identification have been modeled separately in natural language processing. We develop two structured prediction models for joint parsing and multiword expression identification. The first is based on context-free grammars and the second uses tree substitution grammars, a formalism that can store larger syntactic fragments. Our experiments show that both models can identify multiword expressions with much higher accuracy than a state-of-the-art system based on word co-occurrence statistics. We experiment with Arabic and French, which both have pervasive multiword expressions. Relative to English, they also have richer morphology, which induces lexical sparsity in finite corpora. To combat this sparsity, we develop a simple factored lexical representation for the context-free parsing model. Morphological analyses are automatically transformed into rich feature tags that are scored jointly with lexical items. This technique, which we call a factored lexicon, improves both standard parsing and multiword expression identification accuracy.

Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD

In a previous article, we presented a systematic computational study of the extraction of semantic representations from the word-word co-occurrence statistics of large text corpora. The conclusion was that semantic vectors of pointwise mutual information values from very small co-occurrence windows, together with a cosine distance measure, consistently resulted in the best representations across a range of psychologically relevant semantic tasks. This article extends that study by investigating the use of three further factors--namely, the application of stop-lists, word stemming, and dimensionality reduction using singular value decomposition (SVD)--that have been used to provide improved performance elsewhere. It also introduces an additional semantic task and explores the advantages of using a much larger corpus. This leads to the discovery and analysis of improved SVD-based methods for generating semantic representations (that provide new state-of-the-art performance on a standard TOEFL task) and the identification and discussion of problems and misleading results that can arise without a full systematic study.

Exploiting query logs for cross-lingual query suggestions

Query suggestion aims to suggest relevant queries for a given query, which helps users better specify their information needs. Previous work on query suggestion has been limited to the same language. In this article, we extend it to cross-lingual query suggestion (CLQS): for a query in one language, we suggest similar or relevant queries in other languages. This is very important to the scenarios of cross-language information retrieval (CLIR) and other related cross-lingual applications. Instead of relying on existing query translation technologies for CLQS, we present an effective means to map the input query of one language to queries of the other language in the query log. Important monolingual and cross-lingual information such as word translation relations and word co-occurrence statistics, and so on, are used to estimate the cross-lingual query similarity with a discriminative model. Benchmarks show that the resulting CLQS system significantly outperforms a baseline system that uses dictionary-based query translation. Besides, we evaluate CLQS with French-English and Chinese-English CLIR tasks on TREC-6 and NTCIR-4 collections, respectively. The CLIR experiments using typical retrieval models demonstrate that the CLQS-based approach has significantly higher effectiveness than several traditional query translation methods. We find that when combined with pseudo-relevance feedback, the effectiveness of CLIR using CLQS is enhanced for different pairs of languages.

Extracting semantic representations from word co-occurrence statistics: A computational study

The idea that at least some aspects of word meaning can be induced from patterns of word co-occurrence is becoming increasingly popular. However, there is less agreement about the precise computations involved, and the appropriate tests to distinguish between the various possibilities. It is important that the effect of the relevant design choices and parameter values are understood if psychological models using these methods are to be reliably evaluated and compared. In this article, we present a systematic exploration of the principal computational possibilities for formulating and validating representations of word meanings from word co-occurrence statistics. We find that, once we have identified the best procedures, a very simple approach is surprisingly successful and robust over a range of psychologically relevant evaluation measures.

A statistical framework for query translation disambiguation

Resolving ambiguity in the process of query translation is crucial to cross-language information retrieval (CLIR), given the short length of queries. This problem is even more challenging when only a bilingual dictionary is available, which is the focus of our work described here. In this paper, we will present a statistical framework for dictionary-based CLIR that estimates the translation probabilities of query words based on the monolingual word co-occurrence statistics. In addition, we will present two realizations of the proposed framework, i.e., the “maximum coherence model” and the “spectral query-translation model,” that exploit different metrics for the coherence measurement between a translation of a query word and the theme of the entire query. Compared to previous work on dictionary-based CLIR, the proposed framework is advantageous in three aspects: (1) Translation probabilities are calculated explicitly to capture the uncertainty in translating queries; (2) translations of all query words are estimated simultaneously rather than independently; and (3) the formulated problem can be solved efficiently with a unique optimal solution. Empirical studies with Chinese--English cross-language information retrieval using TREC datasets have shown that the proposed models achieve a relative 10%--50% improvement, compared to other approaches that also exploit word co-occurrence statistics for query translation disambiguation.

Explorations in the derivation of word co-occurrence statistics

ABSTRACTRecent work has demonstrated that counts of which other words co-occur with a word of interest can reflect interesting properties of that word. We have studied aspects of this kind of methodology by systematically examining the effects of different combinations of parameters used in the preparation of co-occurrence statistics. Several psychologically relevant evaluation measures are used. We have found that successful performance on the evaluation tasks depends on the correct selection of parameters such as window size and distance metric.