Word Representation With Salient Features

Ming Zhang,Zhicheng Ji,Yan Wang,Vasile Palade

doi:10.1109/access.2019.2892817

Ming Zhang, Zhicheng Ji + Show 2 more

Open Access

https://doi.org/10.1109/access.2019.2892817

Copy DOI

Journal: IEEE Access	Publication Date: Jan 1, 2019
Citations: 35	License type: cc-by

Affiliation: Jiangnan University, Coventry University

Abstract

Inspired from the idea that the contexts in which a word occurs are of different significance, this paper proposes a novel method, called word representation with Salient Features (SaFe), to represent words using salient features selected from the context words. The SaFe method employs the point-wise mutual information (PMI) method with scaled context window to measure word association between a target word and its context. Then, contexts having word associations will be selected as salient features, where the number of salient features for a given word is decided by the ratio between the number of unique contexts and the total counts of occurrences in the whole corpus. The SaFe method can be used with the positive PMI matrix (PPMI), with each row representing a word, hence the name SaFe-PPMI. Moreover, the SaFe-PPMI model can be further decomposed by using the truncated singular vector decomposition technique to obtain dense vectors. In addition to efficient computation, the new models can achieve remarkable improvements in seven semantic relatedness tasks, and they show superior performance when compared with the state-of-the-art models.

Highlights

Semantic relatedness is a metric estimating the degree to which two terms are related, whereas semantic similarity is a subclass of semantic relatedness, which aims to evaluate the likeness of words’ meanings
The performance of the Salient Features (SaFe)-positive PMI matrix (PPMI) model has dramatic improvements on semantic relatedness tasks, and the SaFe-Singular Value Decomposition (SVD) model can significantly outperform some state-of-the-art models, including the Skip-Gram with Negative Sampling method (SGNS) and GloVe models
In this paper, a novel method called SaFe is presented to select salient features for a target word, which is built on the notion that the meaning of a word can be characterized by significant contexts

Summary

Introduction

Semantic relatedness is a metric estimating the degree to which two terms are related, whereas semantic similarity is a subclass of semantic relatedness, which aims to evaluate the likeness of words’ meanings. The underlying factor that determines the efficiency of semantic relatedness measures relies on word representation, which can be classified into two types, i.e., the count-based distributional models and the neural-network-inspired models. Both types of models intrinsically depend on a bag-of-contexts architecture based on the hypothesis that words occurring in the same contexts tend to have similar meanings [6]. The training process of the SGNS method is targeted on individual word-context pairs by predicting the log-probability that a word appears in the context of a given word Another well-known model, called GloVe [15], was trained on the non-zero entries in a global word-context co-occurrence matrix, by optimizing a least squares regression formulation

Methods

Results

Conclusion