Abstract
A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space. For example, ‘gender’ corresponds to v_woman - v_man and v_queen - v_king. This, in turn, allows word analogies to be solved arithmetically: v_king - v_man + v_woman = v_queen. This property is notable because it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models exclusively represent relationships in this manner. We document an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for ‘gender’, we can find an orthogonal matrix R, representing a rotation and reflection, such that R(v_king) = v_queen and R(v_man) = v_woman. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using linear transformations is more accurate than both. Our findings suggest that these transformations can be as good a representation of word relationships as translation vectors.
Highlights
Word embeddings are a cornerstone of current methods in NLP
Gender can be expressed as the translation vectors woman − man and queen − king; past tense can be expressed as thought − think and talked − talk
We find that using orthogonal transformations for analogical reasoning is almost as accurate as using vector arithmetic, and using linear transformations is more accurate than both
Summary
Word embeddings are a cornerstone of current methods in NLP. A notable property of these vectors is that word relationships can exist as linear substructures in the embedding space (Mikolov et al, 2013a). Ethayarajh et al (2019a) proved that when there is no reconstruction error, a word analogy that can be solved arithmetically holds exactly over a set of ordered word pairs iff the co-occurrence shifted PMI is the same for every word pair and across any two word pairs. This means that strict conditions need to be satisfied by the training corpus for a word analogy to hold exactly, and these conditions are not necessarily satisfied by every analogy that makes intuitive sense. Most analogies involving countries and their currency cannot be solved arithmetically using Wikipedia-trained skipgram vectors (Ethayarajh et al, 2019a)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.