Attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines

Fabio Petroni,Jochen L Leidner,Vassilis Plachouras,Timothy Nugent

doi:10.18653/v1/n18-1042

Abstract

Fabio Petroni, Vassilis Plachouras, Timothy Nugent, Jochen L. Leidner. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

Highlights

Neural network-based methods have been successful in advancing the state-of-the-art in a wide range of natural language processing (NLP) tasks, such as dependency parsing (Chen and Manning, 2014), sentence classification (Kim, 2014), machine translation (Sutskever et al, 2014; Luong and Manning, 2016), and information retrieval (Zhang∗work conducted whilst the author was at Thomson Reuters.et al, 2017)
We introduce attr2vec, a novel framework for jointly learning embeddings for words and contextual attributes based on factorization machines
The key advantage of our attr2vec model over GloVe is demonstrated when additional contextual information is considered in the convolutional neural network (CNN) model

Summary

Introduction

Neural network-based methods have been successful in advancing the state-of-the-art in a wide range of NLP tasks, such as dependency parsing (Chen and Manning, 2014), sentence classification (Kim, 2014), machine translation (Sutskever et al, 2014; Luong and Manning, 2016), and information retrieval (Zhang∗work conducted whilst the author was at Thomson Reuters.et al, 2017). The most popular forms of context are neighboring words in a window of text (Mikolov et al, 2013b; Pennington et al, 2014), though examples of additional contextual information might include document topics (Li et al, 2016), dependency relations (Levy and Goldberg, 2014), morphemes (Luong et al, 2013), n-grams (Bojanowski et al, 2017), and sentiment (Tang et al, 2014). The embedding idea was originally devised to help overcome problems associated with the high dimensionality of sparse vector representations of words, in the case of neural network modeling, though embeddings have since been used in a variety of machine learning approaches

Objectives

Methods

Results

Conclusion