Word-level Semantics Research Articles

Mixed inference machine reading comprehension method based on symbolic logic

With the rapid development of machine learning, challenging question and answer datasets have also emerged, and the machine reading comprehension technology has emerged. Traditional machine reading comprehension methods mostly focus on the understanding word level semantics, with the weak ability to extract logical relationships from text, resulting in the lower ability of logical reasoning. In order to strengthen the ability of machine reading comprehension method to extract the logical relationship of text and the ability of logical reasoning, a neural symbol model based on logical reasoning was proposed, and the logical expressions captured by the neural symbol model were converted into text input and trained in a mixed reasoning reading comprehension model based on symbolic logic. The mixed reasoning reading comprehension model based on symbolic logic is different from the traditional machine reading comprehension model. It uses symbolic definition and logical capture to extract logical symbols and generate logical expressions. The research results show that the accuracy and F-measure values of the neural symbol model based on the logical reasoning are 70.08% and 70.05%, respectively, when the training set sample size is 4000. The accuracy of the mixed reasoning reading comprehension model based on symbolic logic in the logical reasoning data set of the standard postgraduate entrance examination is 88.31%, which is higher than the 58.74% of the language perception map network model. The accuracy rate in the four-choice and one-choice question-and-answer data set is 40.92%, which is 1.58% higher than that of the language awareness graph network model. In summary, the neural symbol model and hybrid inference reading comprehension model proposed in the study have superior performance, which can capture the logical relationship of text in data sets well, improve the model feature abstraction and reasoning ability, effectively shorten the training time and improve the model efficiency.

Do we rely on good-enough processing in reading under auditory and visual noise?

Noise, as part of real-life communication flow, degrades the quality of linguistic input and affects language processing. According to predictions of the noisy-channel and good-enough processing models, noise should make comprehenders rely more on word-level semantics instead of actual syntactic relations. However, empirical evidence supporting this prediction is still lacking. For the first time, we investigated whether auditory (three-talker babble) and visual (short idioms appearing next to a target sentence on the screen) noise would trigger greater reliance on semantics and make readers of Russian sentences process the sentences superficially. Our findings suggest that, although Russian speakers generally relied on semantics in sentence comprehension, neither auditory nor visual noise increased this reliance. The only effect of noise on semantic processing was found in reading speed under auditory noise measured by first fixation duration: only without noise, the semantically implausible sentences were read slower than semantically plausible ones. These results do not support the predictions of the study based on the noisy-channel and good-enough processing models, which is discussed in light of the methodological differences among the studies of noise and their possible limitations.

VAST: The Valence-Assessing Semantics Test for Contextualizing Language Models

We introduce VAST, the Valence-Assessing Semantics Test, a novel intrinsic evaluation task for contextualized word embeddings (CWEs). Despite the widespread use of contextualizing language models (LMs), researchers have no intrinsic evaluation task for understanding the semantic quality of CWEs and their unique properties as related to contextualization, the change in the vector representation of a word based on surrounding words; tokenization, the breaking of uncommon words into subcomponents; and LM-specific geometry learned during training. VAST uses valence, the association of a word with pleasantness, to measure the correspondence of word-level LM semantics with widely used human judgments, and examines the effects of contextualization, tokenization, and LM-specific geometry. Because prior research has found that CWEs from OpenAI's 2019 English-language causal LM GPT-2 perform poorly on other intrinsic evaluations, we select GPT-2 as our primary subject, and include results showing that VAST is useful for 7 other LMs, and can be used in 7 languages. GPT-2 results show that the semantics of a word are more similar to the semantics of context in layers closer to model output, such that VAST scores diverge between our contextual settings, ranging from Pearson’s rho of .55 to .77 in layer 11. We also show that multiply tokenized words are not semantically encoded until layer 8, where they achieve Pearson’s rho of .46, indicating the presence of an encoding process for multiply tokenized words which differs from that of singly tokenized words, for which rho is highest in layer 0. We find that a few neurons with values having greater magnitude than the rest mask word-level semantics in GPT-2’s top layer, but that word-level semantics can be recovered by nullifying non-semantic principal components: Pearson’s rho in the top layer improves from .32 to .76. Downstream POS tagging and sentence classification experiments indicate that the GPT-2 uses these principal components for non-semantic purposes, such as to represent sentence-level syntax relevant to next-word prediction. After isolating semantics, we show the utility of VAST for understanding LM semantics via improvements over related work on four word similarity tasks, with a score of .50 on SimLex-999, better than the previous best of .45 for GPT-2. Finally, we show that 8 of 10 WEAT bias tests, which compare differences in word embedding associations between groups of words, exhibit more stereotype-congruent biases after isolating semantics, indicating that non-semantic structures in LMs also mask social biases.

Beyond the Benchmarks: Toward Human-Like Lexical Representations.

To process language in a way that is compatible with human expectations in a communicative interaction, we need computational representations of lexical properties that form the basis of human knowledge of words. In this article, we concentrate on word-level semantics. We discuss key concepts and issues that underlie the scientific understanding of the human lexicon: its richly structured semantic representations, their ready and continual adaptability, and their grounding in crosslinguistically valid conceptualization. We assess the state of the art in natural language processing (NLP) in achieving these identified properties, and suggest ways in which the language sciences can inspire new approaches to their computational instantiation.

BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling

Visual storytelling is a creative and challenging task, aiming to automatically generate a story-like description for a sequence of images. The descriptions generated by previous visual storytelling approaches lack coherence because they use word-level sequence generation methods and do not adequately consider sentence-level dependencies. To tackle this problem, we propose a novel hierarchical visual storytelling framework which separately models sentence-level and word-level semantics. We use the transformer-based BERT to obtain embeddings for sentences and words. We then employ a hierarchical LSTM network: the bottom LSTM receives as input the sentence vector representation from BERT, to learn the dependencies between the sentences corresponding to images, and the top LSTM is responsible for generating the corresponding word vector representations, taking input from the bottom LSTM. Experimental results demonstrate that our model outperforms most closely related baselines under automatic evaluation metrics BLEU and CIDEr, and also show the effectiveness of our method with human evaluation.

Multi-granularity semantic representation model for relation extraction

In natural language, a group of words constitute a phrase and several phrases constitute a sentence. However, existing transformer-based models for sentence-level tasks abstract sentence-level semantics from word-level semantics directly, which override phrase-level semantics so that they may be not favorable for capturing more precise semantics. In order to resolve this problem, we propose a novel multi-granularity semantic representation (MGSR) model for relation extraction. This model can bridge the semantic gap between low-level semantic abstraction and high-level semantic abstraction by learning word-level, phrase-level, and sentence-level multi-granularity semantic representations successively. We segment a sentence into entity chunks and context chunks according to an entity pair. Thus, the sentence is represented as a non-empty segmentation set. The entity chunks are noun phrases, and the context chunks contain the key phrases expressing semantic relations. Then, the MGSR model utilizes inter-word, inner-chunk and inter-chunk three kinds of different self-attention mechanisms, respectively, to learn the multi-granularity semantic representations. The experiments on two standard datasets demonstrate our model outperforms the previous models.

SPARQA: Skeleton-Based Semantic Parsing for Complex Questions over Knowledge Bases

Semantic parsing transforms a natural language question into a formal query over a knowledge base. Many existing methods rely on syntactic parsing like dependencies. However, the accuracy of producing such expressive formalisms is not satisfying on long complex questions. In this paper, we propose a novel skeleton grammar to represent the high-level structure of a complex question. This dedicated coarse-grained formalism with a BERT-based parsing algorithm helps to improve the accuracy of the downstream fine-grained semantic parsing. Besides, to align the structure of a question with the structure of a knowledge base, our multi-strategy method combines sentence-level and word-level semantics. Our approach shows promising performance on several datasets.

Semantic textual similarity between sentences using bilingual word semantics

Semantic textual similarity between sentences is indispensable for many information retrieval tasks. Traditional lexical similarity measures cannot compute the similarity beyond a trivial level. Moreover, they only can capture the textual similarity, but not semantic. In this paper, we propose a method for semantic textual similarity that leverages bilingual word-level semantics to compute the semantic similarity between sentences. To capture word-level semantics, we employ distribute representation of words in two different languages. The similarity function based on the concept-to-concept relationship corresponding to the words is also utilized for the same purpose. Multiple new semantic similarity measures are introduced based on word-embedding models trained on two different corpora in two different languages. Apart from these, another new semantic similarity measure is also introduced using the word sense comparison. The similarity score between the sentences is then computed by applying a linear ranking approach to all proposed measures with their importance score estimated employing a supervised feature selection technique. We conducted experiments on the SemEval Semantic Textual Similarity (STS-2017) test collections. The experimental results demonstrated that our method is effective for measuring semantic textual similarity and outperforms some known related methods.

Learning cross-modal correlations by exploring inter-word semantics and stacked co-attention

Cross-modal information retrieval aims to find heterogeneous data of various modalities from a given query of one modality. The main challenge is to learn the semantic correlations between different modalities and measure the distance across modalities. For text-image retrieval, existing work mostly uses off-the-shelf Convolutional Neural Network (CNN) for image feature extraction. For texts, word-level features such as bag-of-words or word2vec are employed to build deep learning models to represent texts. Besides word-level semantics, the semantic relations between words are also informative but less explored. In this paper, we explore the inter-word semantics by modelling texts by graphs using similarity measure based on word2vec. Besides feature presentations, we further study the problem of information imbalance between different modalities when describing the same semantics. For example textual descriptions often contain more background information that cannot be conveyed by images and vice versa. We propose a stacked co-attention network to progressively learn the mutually attended features of different modalities and enhance their fine-grained correlations. A dual-path neural network is proposed for cross-modal information retrieval. The model is trained by a pairwise similarity loss function to maximize the similarity of relevant text-image pairs and minimize the similarity of irrelevant pairs. Experimental results show that the proposed model outperforms the state-of-the-art methods significantly, with 19% improvement on accuracy for the best case.

A Better Way to Attend: Attention with Trees for Video Question Answering.

We propose a new attention model for video question answering. The main idea of the attention models is to locate on the most informative parts of the visual data. The attention mechanisms are quite popular these days. However, most existing visual attention mechanisms regard the question as a whole. They ignore the word-level semantics where each word can have different attentions and some words need no attention. Neither do they consider the semantic structure of the sentences. Although the Extended Soft Attention (E-SA) model for video question answering leverages the word-level attention, it performs poorly on long question sentences. In this paper, we propose the heterogeneous tree-structured memory network (HTreeMN) for video question answering. Our proposed approach is based upon the syntax parse trees of the question sentences. The HTreeMN treats the words differently where the visual words are processed with an attention module and the verbal ones not. It also utilizes the semantic structure of the sentences by combining the neighbors based on the recursive structure of the parse trees. The understandings of the words and the videos are propagated and merged from leaves to the root. Furthermore, we build a hierarchical attention mechanism to distill the attended features. We evaluate our approach on two datasets. The experimental results show the superiority of our HTreeMN model over the other attention models especially on complex questions.

Oxytocin Modulates Semantic Integration in Speech Comprehension.

Listeners interpret utterances by integrating information from multiple sources including word level semantics and world knowledge. When the semantics of an expression is inconsistent with their knowledge about the world, the listener may have to search through the conceptual space for alternative possible world scenarios that can make the expression more acceptable. Such cognitive exploration requires considerable computational resources and might depend on motivational factors. This study explores whether and how oxytocin, a neuropeptide known to influence social motivation by reducing social anxiety and enhancing affiliative tendencies, can modulate the integration of world knowledge and sentence meanings. The study used a between-participant double-blind randomized placebo-controlled design. Semantic integration, indexed with magnetoencephalography through the N400m marker, was quantified while 45 healthy male participants listened to sentences that were either congruent or incongruent with facts of the world, after receiving intranasally delivered oxytocin or placebo. Compared with congruent sentences, world knowledge incongruent sentences elicited a stronger N400m signal from the left inferior frontal and anterior temporal regions and medial pFC (the N400m effect) in the placebo group. Oxytocin administration significantly attenuated the N400m effect at both sensor and cortical source levels throughout the experiment, in a state-like manner. Additional electrophysiological markers suggest that the absence of the N400m effect in the oxytocin group is unlikely due to the lack of early sensory or semantic processing or a general downregulation of attention. These findings suggest that oxytocin drives listeners to resolve challenges of semantic integration, possibly by promoting the cognitive exploration of alternative possible world scenarios.

Chronological Ordering Based on Context Overlap Detection

Context processing plays an important role in different Natural Language Processing applications. Sentence ordering is one of critical tasks in text generation. Following the same order of sentences in the row sources of text is not necessarily to be applied for the resulted text. Accordingly, a need for chronological sentence ordering is of high importance in this regard. Some researches followed linguistic syntactic analysis and others used statistical approaches. This paper proposes a new model for sentence ordering based on sematic analysis. Word level semantics forms a seed to sentence level sematic relations. The model introduces a clustering technique based on sentences senses relatedness. Following to this, sentences are chronologically ordered through two main steps; overlap detection and chronological cause-effect rules. Overlap detection drills down into each cluster to step through its sentences in chronological sequence. Cause-effect rules forms the linguistic knowledge controlling sentences relations. Evaluation of the proposed algorithm showed the capability of the proposed model to process size free texts, non-domain specific and open to extend the cause-effect rules for specific ordering needs.