Abstract
Artificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures of intrinsic dimensionality and linear decodability of latent variables, and provide mathematical arguments for why such useful predictive representations emerge. We focus throughout on how our results can aid the analysis and interpretation of experimental data.
Highlights
Artificial neural networks have recently achieved many successes in solving sequential processing and planning tasks
We begin with an introductory example of how predictive learning enables the extraction of latent variables characterizing the regularity of transitions among a set of discrete “states”, each of which generates a different observation about the world
We begin by illustrating our core idea— that predictive learning leads neural networks to represent the latent spaces underlying their inputs—in a simple setting
Summary
Artificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit lowdimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. Our goal is to build theoretical and data-analytic tools that explain why a predictive learning process leads to lowdimensional maps of the latent structure of the underlying tasks —and what the general features of such maps in neural recordings might be. This links predictive learning in neural networks with existing mechanisms of extracting latent structure[22,23,24] and low-dimensional representations from data[25]. Our central question is whether a recurrent neural network (RNN) trained on this predictive learning task will extract representations of the underlying low-dimensional latent variables
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have