Competitive spiking and indirect entropy minimization of rate code: Efficient search for hidden components

Abstract

Our motivation, which originates from the psychological and physiological evidences of component-based representations in the brain, is to find neural methods that can efficiently search for structures. Here, an architecture made of coupled parallel working reconstruction subnetworks is presented. Each subnetwork utilizes non-negativity constraint on the generative weights and on the internal representation. ‘Spikes’ are generated within subnetworks via winner take all mechanism. Memory components are modified in order to directly minimize the reconstruction error and to indirectly minimize the entropy of the spike rate distribution, via a combination of a stochastic gradient search and a novel tuning method. This tuning dynamically changes the learning rate: the higher the entropy of the spike rate, the higher the learning rate of the gradient search in the subnetworks. This method effectively reduces the search space and increases the escape probability from high entropy local minima. We demonstrate that one subnetwork can develop localized and oriented components. Coupled networks can discover and sort components into the subnetworks; a problem subject to combinatorial explosion. Synergy between spike code and rate code is discussed.