Non-overlapping Receptive Fields Research Articles

Bridging Global Context Interactions for High-Fidelity Pluralistic Image Completion.

We introduce PICFormer, a novel framework for Pluralistic Image Completion using a transFormer based architecture, that achieves both high quality and diversity at a much faster inference speed. Our key contribution is to introduce a code-shared codebook learning using a restrictive CNN on small and non-overlapping receptive fields (RFs) for the local visible token representation. This results in a compact yet expressive discrete representation, facilitating efficient modeling of global visible context relations by the transformer. Unlike the prevailing autoregressive approaches, we proposed to sample all tokens simultaneously, leading to more than 100× faster inference speed. To enhance appearance consistency between visible and generated regions, we further propose a novel attention-aware layer (AAL), designed to better exploit distantly related high-frequency features. Through extensive experiments, we demonstrate that the efficiently learns semantically-rich discrete codes, resulting in significantly improved image quality. Moreover, our diverse image completion framework surpasses state-of-the-art methods on multiple image completion datasets. The project page is available at https://chuanxiaz.com/picformer/.

Spatial Attention Modulates Spike Count Correlations and Granger Causality in the Primary Visual Cortex.

The influence of spatial attention on neural interactions has been revealed even in early visual information processing stages. It resolves the process of competing for sensory information about objects perceived as targets and distractors. However, the attentional modulation of the interaction between pairs of neurons with non-overlapping receptive fields (RFs) is not well known. Here, we investigated the activity of anatomically distant neurons in two behaving monkeys’ primary visual cortex (V1), when they performed a spatial attention task detecting color change. We compared attentional modulation from the perspective of spike count correlations and Granger causality among simple and complex cells. An attention-related increase in spike count correlations and a decrease in Granger causality were found. The results showed that spatial attention significantly influenced only the interactions between rather than within simple and complex cells. Furthermore, we found that the attentional modulation of neuronal interactions changed with neuronal pairs’ preferred directions differences. Thus, we found that spatial attention increased the functional communications and competing connectivities when attending to the neurons’ RFs, which impacts the interactions only between simple and complex cells. Our findings enrich the model of simple and complex cells and further understand the way that attention influences the neurons’ activities.

Multistable network dynamics through lateral inhibition: an efficient mechanism for selective information routing

A remarkable feature of the brain is its ability to flexibly process information in a context-dependent manner. A hallmark example is selective visual attention. It allows to focus on relevant information, and to ignore distracting features of a visual scene, effectively routing information through the visual hierarchy [1]. A seminal hypothesis about how this routing is accomplished is 'Communication through Coherence' (CTC) [2]: A neuron with oscillating membrane potential can be driven to fire by a relatively weak input when it is close to the threshold, while if it is in a phase of low excitability, even stronger input might fail to elicit an action potential. Thus, shifting relative phases of oscillating sending and receiving populations could be utilized to gate information. Here, we investigate neural mechanisms of information routing and their effectiveness in a generic, two-layered cortical network model with lateral recurrent interactions. Building on the CTC hypothesis, we construct a biophysically plausible network model mimicking the fan-in structure of the visual processing stream. It comprises four populations A, B, C and D of recurrently coupled networks of conductance-based spiking neurons. A and B correspond to sending populations in a lower visual area and have non-overlapping receptive fields. They project to receiving populations C and D in a higher visual area with larger receptive fields. Upon activation, all populations engage in oscillatory activity. We show that introducing distance-dependent lateral inhibition brings the receiving populations in phase, and the sending populations in anti-phase, thus establishing a phase competition between the two sending populations: Only one population can be in a favorable in-phase relation with the receiving populations, since the other is automatically forced into anti-phase, thus hindering information transfer. This 'self-organization' of phase relations mediated by lateral inhibition distinguishes our approach from previous modeling efforts (e.g. [3-5]). When increasing the strength of the lateral inhibition between A and B, the system transitions from a regime of mixed representations to a bistable one, where one sending population is in a favorable in-phase relation for an extended period of time. In this regime, a brief current pulse to one sending population is sufficient to set a desired phase relation and thus 'switch' information routing in the network. This mechanism bears two interesting consequences: First, it allows neural circuits to control routing using energy-efficient, short bursts of neural activity instead of requiring sustained elevation of firing rates. Second, such pulses delivered by electrical stimulation could be employed to externally direct the focus of attention, a possible application in the field of brain prosthetics.

Spike synchronization in cat primary visual cortex depends on similarity of surround‐suppression magnitude

In the primary visual cortex (V1), the spike synchronization seen in neuron pairs with non-overlapping receptive fields can be explained by similarities in their preferred orientation (PO). However, this is not true for pairs with overlapping receptive fields, as they can still exhibit spike synchronization even if their POs are only weakly correlated. Here, we investigated the relationship between spike synchronization and suppressive modulation derived from classical receptive-field surround (surround suppression). We found that layer 4 and layer 2/3 pairs exhibited mainly asymmetric spike synchronization that had non-zero time-lags and was dependent on both the similarity of the PO and the strength of surround suppression. In contrast, layer 2/3 and layer 2/3 pairs showed mainly symmetric spike synchronization that had zero time-lag and was dependent on the similarity of the strength of surround suppression but not on the similarity in POs. From these results, we propose that in cat V1 there exists a functional network that mainly depends on the similarity in surround suppression, and that in layer 2/3 neurons the network maintains surround suppression that is primarily inherited from layer 4 neurons.

A model for selective visual attention predicts information routing

A remarkable feature of the brain is its ability to flexibly process information in a context-dependent manner. For example, selective attention allows to focus on relevant information, and to ignore distracting features of a visual scene. Here we develop a model to investigate a putative neural mechanism by which attention might dynamically change the functional configuration of networks in visual cortex. Since efferents from multiple neurons in an early visual area converge onto the same neuron in a higher area, synaptic input induced by an attended object should be more efficient in driving the post-synaptic neuron than an input induced by a simultaneously presented, but unattended object. In fact, it has been found that the firing rate of a post-synaptic neuron in such a situation is similar to its activity when the unattended stimulus is absent. However, if no attention is directed while two stimuli are present, the firing rate of the post-synaptic neuron is approximately at the mean of the rates at which it would fire if each of two stimuli were presented alone (biased competition [1,2]). Attention is thus assumed to selectively promote information transfer between visual areas. Recent experiments support this idea by showing that V4 local field potentials are correlated with a temporal modulation of the attended stimulus and not with the temporal modulation of a non-attended stimulus in the same receptive field (gating [3]). Several models were proposed to explain the effect of biased competition and information routing [4,5]. However, here we construct a biophysically plausible model that specifically reproduces both experimental results within the same framework. It comprises four populations of recurrently coupled networks of conductance-based spiking neurons with lateral inhibition. Upon activation, these populations engage in oscillatory activity. The first two networks correspond to a lower visual area with non-overlapping receptive fields, which both project to the third and the fourth network corresponding to higher area populations with larger receptive fields. We assume attention to manifest as an additional input to one lower area population. Deployment of attention consequently results in a bias of firing rates between the populations in the lower area. This bias induces a phase relation between the lower and the higher area that is more suitable for information transmission from the attended stimulus. We show that this mechanism is capable to explain both, the biased competition effect of the firing rate, and selective gating of information flow. We thus propose a combination of cortical network topology, neuronal population dynamics, and selective additional input as putative neural underpinnings of attention.

Processing Spatial Relations With Different Apertures of Attention

Neuropsychological studies suggest the existence of lateralized networks that represent categorical and coordinate types of spatial information. In addition, studies with neural networks have shown that they encode more effectively categorical spatial judgments or coordinate spatial judgments, if their input is based, respectively, on units with relatively small, nonoverlapping receptive fields, as opposed to units with relatively large, overlapping receptive fields. These findings leave open the question of whether interactive processes between spatial detectors and types of spatial relations can be modulated by spatial attention. We hypothesized that spreading the attention window to encompass an area that includes two objects promotes coordinate spatial relations, based on coarse coding by large, overlapping, receptive fields. In contrast, narrowing attention to encompass an area that includes only one of the objects benefits categorical spatial relations, by effectively parsing space. By use of a cueing procedure, the spatial attention window was manipulated to select regions of differing areas. As predicted, when the attention window was large, coordinate spatial transformations were noticed faster than categorical transformations; in contrast, when the attention window was relatively smaller, categorical spatial transformations were noticed faster than coordinate transformations. Another novel finding was that coordinate changes were noticed faster when cueing an area that included both objects as well as the empty space between them than when simultaneously cueing both areas including the objects while leaving the gap between them uncued.

Relationship between spontaneous and evoked spike-time correlations in primate visual cortex.

Coincident spikes have been implicated in vision-related processes such as feature binding, gain modulation, and long-distance communication. The source of these spike-time correlations is unknown. Although several studies have proposed that cortical spikes are correlated based on stimulus structure, others have suggested that spike-time correlations reflect ongoing cortical activity present even in the absence of a coherent visual stimulus. To examine this issue, we collected single-unit recordings from primary visual cortex (V1) of the anesthetized and paralyzed prosimian bush baby using a 100-electrode array. Spike-time correlations for pairs of cells were compared under three conditions: a moving grating at the cells' preferred orientation, an equiluminant blank screen, and a dark condition with eyes covered. The amplitudes, lags, and widths of cross-correlation histograms (CCHs) were strongly correlated between these conditions although for the blank stimulus and dark condition, the CCHs were broader with peaks lower in amplitude. In both preferred stimulus and blank conditions, the CCH amplitudes were greater when the cells within the pair had overlapping receptive fields and preferred similar orientations rather than nonoverlapping receptive fields and different orientations. These data suggest that spike-time correlations present in evoked activity are generated by mechanisms common to those operating in spontaneous conditions.

Neural representations of visual salience in primary visual cortex

The visual stimuli of our experiment consisted of an array of randomly drifting Gabor patches, with a subset aligned to form a coherently drifting closed contour. This contour was designed to optimally stimulate the non-overlapping receptive fields of two or more neurons under study (high salience condition). Two additional salience conditions (middle and low), were tested by adding orientation jitter to the Gabor patches that were not stimulating the receptive fields of the recorded cells. Contours also appeared with equal probability at an alternative control location that was equidistant from the fixation point. (Fig. 1) Figure 1. Stimuli used in the contour detection task. The diagrams in A and D illustrate how the figure and background patches were positioned relative to the cellular receptive fields (solid white boxes) and the center of gaze (white dot). The figures in B and E have full motion coherence and high salience, while the figures in C and F include ±30 degrees of o r ien ta t ion j i t t e r. Fo r simplicity, the arrows for all of the background patches have been excluded.

Correlated neural activity as the driving force for functional changes in auditory cortex

The functional role of neural synchrony is reflected in cortical tonotopic map reorganization and in the emergence of pathological phenomena such as tinnitus. First of all experimenter-centered and subject-centered views of neural activity will be contrasted; this argues against the use of stimulus-correction procedures and favors the use of a correction procedure based on neural activity without reference to stimulus timing. Within a cortical column neurons fired synchronously with on average about 6% of their spikes in a 1 ms bin and occasionally showing 30% or more of such coincident spikes. For electrode separations exceeding 200 μm the average peak correlation strength only occasionally reached 3%. The experimental evidence for coincidence of neural activity, neural correlation and neural synchrony shows that horizontal fibers activity can induce strong neural correlations. Cortico-cortical connections for a large part connect cell groups with characteristic frequencies differing by more than one octave. Such neurons have generally non-overlapping receptive fields but still can have sizeable peak cross-correlations. Correlated neural activity and heterotopic neural interconnections are presented as the substrates for cortical reorganization; increased neural synchrony and tonotopic map reorganization go hand in hand. This links cortical reorganization with hypersynchrony that can be considered as an important driving force underlying tinnitus.

Does Neuronal Synchrony Underlie Visual Feature Grouping?

Previous research suggests that synchronous neural activity underlies perceptual grouping of visual image features. The generality of this mechanism is unclear, however, as previous studies have focused on pairs of neurons with overlapping or collinear receptive fields. By sampling more broadly and employing stimuli that contain partially occluded objects, we have conducted a more incisive test of the binding by synchrony hypothesis in area MT. We find that synchrony in spiking activity shows little dependence on feature grouping, whereas gamma band synchrony in field potentials can be significantly stronger when features are grouped. However, these changes in gamma band synchrony are small relative to the variability of synchrony across recording sites and do not provide a robust population signal for feature grouping. Moreover, these effects are reduced when stimulus differences nearby the receptive fields are eliminated using partial occlusion. Our findings suggest that synchrony does not constitute a general mechanism of visual feature binding.

Learning with two sites of synaptic integration

Since the classical work of D O Hebb 1949 The Organization of Behaviour (New York: Wiley) it is assumed that synaptic plasticity solely depends on the activity of the pre- and the post-synaptic cells. Synapses influence the plasticity of other synapses exclusively via the post-synaptic activity. This confounds effects on synaptic plasticity and neuronal activation and, thus, makes it difficult to implement networks which optimize global measures of performance. Exploring solutions to this problem, inspired by recent research on the properties of apical dendrites, we examine a network of neurons with two sites of synaptic integration. These communicate in such a way that one set of synapses mainly influences the neurons' activity; the other set gates synaptic plasticity. Analysing the system with a constant set of parameters reveals: (1) the afferents that gate plasticity act as supervisors, individual to every cell. (2) While the neurons acquire specific receptive fields the net activity remains constant for different stimuli. This ensures that all stimuli are represented and, thus, contributes to information maximization. (3) Mechanisms for maximization of coherent information can easily be implemented. Neurons with non-overlapping receptive fields learn to fire correlated and preferentially transmit information that is correlated over space. (4) We demonstrate how a new measure of performance can be implemented: cells learn to represent only the part of the input that is relevant to the processing at higher stages. This criterion is termed ‘relevant infomax’.

Correlations between hidden units in multilayer neural networks and replica symmetry breaking.

We consider feed-forward neural networks with one hidden layer, tree architecture, and a fixed hidden-to-output Boolean function. Focusing on the saturation limit of the storage problem the influence of replica symmetry breaking on the distribution of local fields at the hidden units is investigated. These field distributions determine the probability of finding a specific activation pattern of the hidden units as well as the corresponding correlation coefficients and therefore quantify the division of labor among the hidden units. We find that although modifying the storage capacity and the distribution of local fields markedly replica symmetry breaking has only a minor effect on the correlation coefficients. Detailed numerical results are provided for the PARITY, COMMITTEE, and AND machines with K=3 hidden units and nonoverlapping receptive fields.

Phase transitions in the generalization behaviour of multilayer perceptrons: II. The influence of noise

We extend our study of phase transitions in the generalization behaviour of multilayer perceptrons with non-overlapping receptive fields to the problem of the influence of noise, concerning e.g. the input units and/or the couplings between the input units and the hidden units of the second layer (='input noise'), or the final output unit (='output noise'). Without output noise, the output itself is given by a general, permutation-invariant Boolean function of the outputs of the hidden units. As a result we find that the phase transitions, which we found in the deterministic case, mostly persist in the presence of noise. The influence of the noise on the position of the phase transition, as well as on the behaviour in other regimes of the loading parameter $\alpha$, can often be described by a simple rescaling of $\alpha$ depending on strength and type of the noise. We then consider the problem of the optimal noise level for Gibbsian and Bayesian learning, looking on replica symmetry breaking as well. Finally we consider the question why learning with errors is useful at all.

Storage capacities of committee machines with overlapping and non-overlapping receptive fields

We present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number M of hidden units and spherical weights. Difficulties arise in the solution of this problem in the limit of large M. In the case of overlapping receptive fields, as the number of patterns increases, both permutation symmetry and replica symmetry are broken, which leads to the appearance of many order parameters and causes additional difficulty. We observe that the relations among these order parameters yield a set of quantities which are small in the limit of large M, making the asymptotic calculation tractable. Using the one-step replica symmetry breaking scheme, we compute the asymptotic value of the storage capacity per input unit in the limit of large M. We find that . The shift to the case of non-overlapping receptive fields can be made easily; we then find . Both values satisfy the bound of Mitchison and Durbin.

Configuration Saliency Revealed by Dichoptic Masking

We studied suprathreshold spatial integration by testing the saliency of multi Gabor element configurations in dichoptic masking conditions. Dichoptic presentations allows for a competition between spatially overlapping suprathreshold stimuli that involve nonoverlapping receptive fields in the first stage of visual filtering. Different spatial configurations of Gabor patches (sigma=lambda=0.12 deg) were presented to one eye (target) together with a bandpass noise presented to the other eye (mask). After a short rivalry period (128 ms) in which a dominance of one eye was established, a probe (a randomly positioned small rectangle of reduced contrast in the target) was presented for an additional detection period (80 ms). Probe detection performance was measured (2AFC) by finding the mask contrast leading to 75% correct response. We find that textures of randomly oriented patches are more dominant than uniform textures where the effect decreases and even reverses with decreasing of contrast. For suprathreshold (1-D) contours, however, we find that smooth collinear contours are more dominant than ‘jagged’ ones, regardless of contrast. These findings suggest principles underlying an early lateral integration mechanism based on contrast-dependent inhibitory and excitatory connections. This mechanism could be based on iso-orientation surround (2-D) inhibition and collinear (1-D) facilitation, with inhibition being more effective at high contrasts.

Stimulus-dependent modulations of correlated high-frequency oscillations in cat visual cortex.

The hypothesis that correlated neural activity is involved in the cortical representation of visual stimuli was examined by recording multi-unit activity and local field potentials from neurons with non-overlapping receptive fields in areas 17 and 18. Using coherence functions, correlations of oscillatory patterns (35-100 Hz) of neural signals were investigated under three stimulus conditions: (i) a whole field grating or a long bar moving across both receptive fields; (ii) masking the region between both receptive fields while stimulating the remaining visual field; and (iii) two separate stimuli simultaneously moving in opposite directions. Coherences of oscillations were found to be significantly higher in the first stimulus condition than in the other two conditions. Since different visual stimuli were reflected in the coherence of neural activity, we concluded that correlated neural activity is a potential candidate for coding of sensory information.

Encoding shape and spatial relations: The role of receptive field size in coordinating complementary representations

An effective functional architecture facilitates interactions among subsystems that are often used together. Computer simulations showed that differences in receptive field sizes can promote such organization. When input was filtered through relatively small nonoverlapping receptive fields, artificial neural net-works learned to categorize shapes relatively quickly; in contrast, when input was filtered through relatively large overlapping receptive fields, networks learned to encode specific shape exemplars or metric spatial relations relatively quickly. Moreover, when the receptive field sizes were allowed to adapt during learning, networks developed smaller receptive fields when they were trained to categorize shapes or spatial relations, and developed larger receptive fields when they were trained to encode specific exemplars or metric distances. In addition, when pairs of networks were constrained to use input from the same type of receptive fields, networks learned a task faster when they were paired with networks that were trained to perform a compatible type of task. Finally, using a novel modular architecture, networks were not preassigned a task, but rather competed to perform the different tasks. Networks with small nonover-lapping receptive fields tended to win the competition for categorical tasks whereas networks with large overlapping receptive fields tended to win the competition for exemplar/metric tasks.

Computational capabilities of restricted two-layered perceptrons.

We study the extent to which fixing the second-layer weights reduces the capacity and generalization ability of a two-layer perceptron. Architectures with N inputs, K hidden units, and a single output are considered, with both overlapping and nonoverlapping receptive fields. We obtain from simulations one measure of the strength of a network---its critical capacity, ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$. Using the ansatz ${\mathrm{\ensuremath{\tau}}}_{\mathrm{med}}$\ensuremath{\propto}(${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$-\ensuremath{\alpha}${)}^{\mathrm{\ensuremath{-}}2}$ to describe the manner in which the median learning time diverges as ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ is approached, we estimate ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ in a manner that does not depend on arbitrary impatience parameters. The chir learning algorithm is used in our simulations. For K=3 and overlapping receptive fields we show that the general machine is equivalent to the committee machine with the same architecture. For K=5 and the same connectivity the general machine is the union of four distinct networks with fixed second layer weights, of which the committee machine is the one with the highest ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$. Since the capacity of the union of a finite set of machines equals that of the strongest constituent, the capacity of the general machine with K=5 equals that of the committee machine. We were not able to prove this for general K , but believe that it does hold. We investigated the internal representations used by different machines, and found that high correlations between the hidden units and the output reduce the capacity. Finally we studied the Boolean functions that can be realized by networks with fixed second layer weights. We discovered that two different machines implement two completely distinct sets of Boolean functions.

Encoding Shape and Spatial Relations: The Role of Receptive Field Size in Coordinating Complementary Representations

An effective functional architecture facilitates interactions among subsystems that are often used together. Computer simulations showed that differences in receptive field sizes can promote such organization. When input was filtered through relatively small nonoverlapping receptive fields, artificial neural net‐works learned to categorize shapes relatively quickly; in contrast, when input was filtered through relatively large overlapping receptive fields, networks learned to encode specific shape exemplars or metric spatial relations relatively quickly. Moreover, when the receptive field sizes were allowed to adapt during learning, networks developed smaller receptive fields when they were trained to categorize shapes or spatial relations, and developed larger receptive fields when they were trained to encode specific exemplars or metric distances. In addition, when pairs of networks were constrained to use input from the same type of receptive fields, networks learned a task faster when they were paired with networks that were trained to perform a compatible type of task. Finally, using a novel modular architecture, networks were not preassigned a task, but rather competed to perform the different tasks. Networks with small nonover‐lapping receptive fields tended to win the competition for categorical tasks whereas networks with large overlapping receptive fields tended to win the competition for exemplar/metric tasks.

Perfect loss of generalization due to noise in K=2 parity machines

Learning in a specific type of multilayer network referred to as a K=2 parity machine is studied in the limit that both the system size N and the number of examples m become infinite while the ratio alpha =m/N remains finite. The machine consists of K=2 hidden units with non-overlapping receptive fields each of size N/2. The output is the sign of the product of the two hidden units for each input We investigate incremental learning by empirically using a least-action algorithm in the following two learning paradigms. In the first, it is assumed that each example is transmitted perfectly to a student. We show that an ability to generalize emerges as the rescaled length of the connection vector l reaches a critical value lc. Further, we show that a student can identify the target exactly in the limit alpha to infinity , where the prediction error epsilon decreases to zero as epsilon approximately 0.441 alpha -13/. In the second paradigm, we examine what happens if each teacher signal is reversed to the opposite sign at a noise rate l. For small lambda , it is found that the prediction error converges to a finite value of O( square root lambda ) in O( lambda -32/) iterations. However, for a noise rate beyond a critical value lambda c approximately 0.175, the student cannot acquire any generalization ability even as alpha to infinity .