Memory capacity of networks with stochastic binary synapses.

Alexis M Dubreuil,Yali Amit,Nicolas Brunel,Claus C Hilgetag

doi:10.1371/journal.pcbi.1003727

Abstract

In standard attractor neural network models, specific patterns of activity are stored in the synaptic matrix, so that they become fixed point attractors of the network dynamics. The storage capacity of such networks has been quantified in two ways: the maximal number of patterns that can be stored, and the stored information measured in bits per synapse. In this paper, we compute both quantities in fully connected networks of N binary neurons with binary synapses, storing patterns with coding level , in the large and sparse coding limits (). We also derive finite-size corrections that accurately reproduce the results of simulations in networks of tens of thousands of neurons. These methods are applied to three different scenarios: (1) the classic Willshaw model, (2) networks with stochastic learning in which patterns are shown only once (one shot learning), (3) networks with stochastic learning in which patterns are shown multiple times. The storage capacities are optimized over network parameters, which allows us to compare the performance of the different models. We show that finite-size effects strongly reduce the capacity, even for networks of realistic sizes. We discuss the implications of these results for memory storage in the hippocampus and cerebral cortex.

Highlights

Attractor neural networks have been proposed as long-term memory storage devices [1,2,3]
Two central hypotheses in neuroscience are that long-term memory is sustained by modifications of the connectivity of neural circuits, while short-term memory is sustained by persistent neuronal activity following the presentation of a stimulus
They have been implemented in attractor network models, that store specific patterns of activity using Hebbian plasticity rules, which allow retrieval of these patterns as attractors of the network dynamics

Summary

Introduction

Attractor neural networks have been proposed as long-term memory storage devices [1,2,3] In such networks, a pattern of activity (the set of firing rates of all neurons in the network) is said to be memorized if it is one of the stable states of the network dynamics. In the sparse coding limit (in which the average fraction of selective neurons per pattern f goes to zero in the large N limit), the capacity was shown to diverge as 1=(f D log (f )D) These scalings lead to a network storing on the order of 1 bit per synapse, in the large N limit, for any value of the coding level. Elizabeth Gardner [10] computed the maximal capacity, in the space of all possible coupling matrices, and demonstrated a similar scaling for capacity and information stored per synapse

Objectives

Methods

Results

Discussion

Conclusion