Abstract
Natural language is inherently a discrete symbolic representation of human knowledge. Recent advances in machine learning (ML) and in natural language processing (NLP) seem to contradict the above intuition: discrete symbols are fading away, erased by vectors or tensors called distributed and distributional representations. However, there is a strict link between distributed/distributional representations and discrete symbols, being the first an approximation of the second. A clearer understanding of the strict link between distributed/distributional representations and symbols may certainly lead to radically new deep learning networks. In this paper we make a survey that aims to renew the link between symbolic representations and distributed/distributional representations. This is the right time to revitalize the area of interpreting how discrete symbols are represented inside neural networks.
Highlights
Natural language is inherently a discrete symbolic representation of human knowledge
In the rest of the section, we present how to build matrices representing words in context, we will shortly recap on how dimensionality reduction techniques have been used in distributional semantics, and, we report on word2vec (Mikolov et al, 2013), which is a novel distributional semantic techniques based on deep learning
In the ‘90, the hot debate on neural networks was whether or not distribute representations are only an implementation of discrete symbolic representations. The question behind this debate is crucial to understand if neural networks may exploit something more that systems strictly based on discrete symbolic representations
Summary
Natural language is inherently a discrete symbolic representation of human knowledge. Model interpretability is becoming an important topic in machine learning in general (Lipton, 2018) This clearer understanding is the dawn of a new range of possibilities: understanding what part of the current symbolic techniques for natural language processing have a sufficient representation in deep neural networks; and, understanding whether a more brain-like model—the neural networks—is compatible with methods for syntactic parsing or semantic processing that have been defined in these decades of studies in computational linguistics and natural language processing. In this paper we make a survey that aims to draw the link between symbolic representations and distributed/distributional representations This is the right time to revitalize the area of interpreting how symbols are represented inside neural networks. We discuss more in depth the general issue of compositionality, analyzing three different approaches to the problem: compositional distributional semantics (Clark et al, 2008; Baroni et al, 2014), holographic reduced representations (Plate, 1994; Neumann, 2001), and recurrent neural networks (Socher et al, 2012; Kalchbrenner and Blunsom, 2013)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
