Border Ownership Research Articles

Border-ownership tuning determines the connectivity between V4 and V1 in the macaque visual system

Cortical feedback connections are extremely numerous but the logic of connectivity between higher and lower areas remains poorly understood. Feedback from higher visual areas to primary visual cortex (V1) has been shown to enhance responses on perceptual figures compared to backgrounds, an effect known as figure-background modulation (FBM). A likely source of this feedback are border-ownership (BO) selective cells in mid-tier visual areas (e.g. V4) which represent the location of figures. We examined the connectivity between V4 cells and V1 cells using noise-correlations and micro-stimulation to estimate connectivity strength. We show that connectivity is consistent with a model in which BO-tuned V4 cells send positive feedback in the direction of their preferred figure and negative feedback in the opposite direction. This connectivity scheme can recreate patterns of FBM observed in previous studies. These results provide insights into the cortical connectivity underlying figure-background perception and establish a link between FBM and BO-tuning.

Dependence of Contextual Modulation in Macaque V1 on Interlaminar Signal Flow.

In visual cortex, neural correlates of subjective perception can be generated by modulation of activity from beyond the classical receptive field (CRF). In macaque V1, activity generated by nonclassical receptive field (nCRF) stimulation involves different intracortical circuitry than activity generated by CRF stimulation, suggesting that interactions between neurons across V1 layers differ under CRF and nCRF stimulus conditions. Using Neuropixels probes, we measured border ownership modulation within large, local populations of V1 neurons. We found that neurons in single columns preferred the same side of objects located outside of the CRF. In addition, we found that cross-correlations between pairs of neurons situated across feedback/horizontal and input layers differed between CRF and nCRF stimulation. Furthermore, independent of the comparison with CRF stimulation, we observed that the magnitude of border ownership modulation increased with the proportion of information flow from feedback/horizontal layers to input layers. These results demonstrate that the flow of signals between layers covaries with the degree to which neurons integrate information from beyond the CRF.

Border ownership signals emerge in an artificial neural network trained to predict future visual input

Border ownership signals emerge in an artificial neural network trained to predict future visual input

Brain-like border ownership signals support prediction of natural videos.

To make sense of visual scenes, the brain must segment foreground from background. This is thought to be facilitated by neurons in the primate visual system that encode border ownership (BOS), i.e. whether a local border is part of an object on one or the other side of the border. It is unclear how these signals emerge in neural networks without a teaching signal of what is foreground and background. In this study, we investigated whether BOS signals exist in PredNet, a self-supervised artificial neural network trained to predict the next image frame of natural video sequences. We found that a significant number of units in PredNet are selective for BOS. Moreover these units share several other properties with the BOS neurons in the brain, including robustness to scene variations that constitute common object transformations in natural videos, and hysteresis of BOS signals. Finally, we performed ablation experiments and found that BOS units contribute more to prediction than non-BOS units for videos with moving objects. Our findings indicate that BOS units are especially useful to predict future input in natural videos, even when networks are not required to segment foreground from background. This suggests that BOS neurons in the brain might be the result of evolutionary or developmental pressure to predict future input in natural, complex dynamic visual environments.

Visual cortical processing—From image to object representation

Image understanding is often conceived as a hierarchical process with many levels, where complexity and invariance of object representation gradually increase with level in the hierarchy. In contrast, neurophysiological studies have shown that figure-ground organization and border ownership coding, which imply understanding of the object structure of an image, occur at levels as low as V1 and V2 of the visual cortex. This cannot be the result of back-projections from object recognition centers because border-ownership signals appear well-before shape selective responses emerge in inferotemporal cortex. Ultra-fast border-ownership signals have been found not only for simple figure displays, but also for complex natural scenes. In this paper I review neurophysiological evidence for the hypothesis that the brain uses dedicated grouping mechanisms early on to link elementary features to larger entities we might call “proto-objects”, a process that is pre-attentive and does not rely on object recognition. The proto-object structures enable the system to individuate objects and provide permanence, to track moving objects and cope with the displacements caused by eye movements, and to select one object out of many and scrutinize the selected object. I sketch a novel experimental paradigm for identifying grouping circuits, describe a first application targeting area V4, which yielded negative results, and suggest targets for future applications of this paradigm.

Disinhibitory circuit mediated by connections from vasoactive intestinal polypeptide to somatostatin interneurons underlies the paradoxical decrease in spike synchrony with increased border ownership selective neuron firing rate.

The activity of border ownership selective (BOS) neurons in intermediate-level visual areas indicates which side of a contour owns a border relative to its classical receptive field and provides a fundamental component of figure-ground segregation. A physiological study reported that selective attention facilitates the activity of BOS neurons with a consistent border ownership preference, defined as two neurons tuned to respond to the same visual object. However, spike synchrony between this pair is significantly suppressed by selective attention. These neurophysiological findings are derived from a biologically-plausible microcircuit model consisting of spiking neurons including two subtypes of inhibitory interneurons, somatostatin (SOM) and vasoactive intestinal polypeptide (VIP) interneurons, and excitatory BOS model neurons. In our proposed model, BOS neurons and SOM interneurons cooperate and interact with each other. VIP interneurons not only suppress SOM interneuron responses but also are activated by feedback signals mediating selective attention, which leads to disinhibition of BOS neurons when they are directing selective attention toward an object. Our results suggest that disinhibition arising from the synaptic connections from VIP to SOM interneurons plays a critical role in attentional modulation of neurons in intermediate-level visual areas.

Functional recursion of orientation cues in figure-ground separation

Visual texture is an important cue to figure-ground organization. While processing of texture differences is a prerequisite for the use of this cue to extract figure-ground organization, these stages are distinct processes. One potential indicator of this distinction is the possibility that texture statistics play a different role in the figure vs. in the ground. To determine whether this is the case, we probed figure-ground processing with a family of local image statistics that specified textures that varied in the strength and spatial scale of structure, and the extent to which features are oriented. For image statistics that generated approximately isotropic textures, the threshold for identification of figure-ground structure was determined by the difference in correlation strength in figure vs. ground, independent of whether the correlations were present in figure, ground, or both. However, for image statistics with strong orientation content, thresholds were up to two times higher for correlations in the ground, vs. the figure. This held equally for texture-defined objects with convex or concave boundaries, indicating that these threshold differences are driven by border ownership, not boundary shape. Similar threshold differences were found for presentation times ranging from 125 to 500 ms. These findings identify a qualitative difference in how texture is used for figure-ground analysis, vs. texture discrimination. Additionally, it reveals a functional recursion: texture differences are needed to identify tentative boundaries and consequent scene organization into figure and ground, but then scene organization modifies sensitivity to texture differences according to the figure-ground assignment.

Columnar processing of border ownership in primate visual cortex.

To understand a visual scene, the brain segregates figures from background by assigning borders to foreground objects. Neurons in primate visual cortex encode which object owns a border (border ownership), but the underlying circuitry is not understood. Here, we used multielectrode probes to record from border ownership-selective units in different layers in macaque visual area V4 to study the laminar organization and timing of border ownership selectivity. We find that border ownership selectivity occurs first in deep layer units, in contrast to spike latency for small stimuli in the classical receptive field. Units on the same penetration typically share the preferred side of border ownership, also across layers, similar to orientation preference. Units are often border ownership-selective for a range of border orientations, where the preferred sides of border ownership are systematically organized in visual space. Together our data reveal a columnar organization of border ownership in V4 where the earliest border ownership signals are not simply inherited from upstream areas, but computed by neurons in deep layers, and may thus be part of signals fed back to upstream cortical areas or the oculomotor system early after stimulus onset. The finding that preferred border ownership is clustered and can cover a wide range of spatially contiguous locations suggests that the asymmetric context integrated by these neurons is provided in a systematically clustered manner, possibly through corticocortical feedback and horizontal connections.

The orientation-difference cue in figure-ground separation: border ownership and timing

Figure-ground separation, a crucial component of visual processing, can be driven by differences in contrast, orientation, and other local cues. We previously showed (VSS 2020) that the orientation cue has distinctive characteristics: figure-ground separation depends not only on the magnitude of the orientation difference, but also on whether the oriented component is present in the figure vs. the ground. Here we show that this dependence is invariant with respect to convexity vs. concavity of the figure, and determine how it evolves over time. In a 2-AFC task, subjects (N=3) distinguished a target image containing five randomly-positioned figures rendered by one synthetic texture superimposed on a background defined by a second synthetic texture, from a non-target image consisting of a uniform texture whose image statistics matched the spatial average of the target’s statistics. Target and non-target images consisted of 64 x 64 arrays of 10-min black and white checks, and the figures covered 25% of the target’s area. Texture construction enabled control of the spatial frequency content of figure and ground in one or two orientations. For figures consisting of circular discs, figure-ground separation was facilitated when the figure texture was oriented, compared to when the ground was oriented. This difference was linked to the degree of anisotropy, as determined from the textures’ spectra (r=0.82, p<0.001). When figure boundaries were changed from convex to concave, results were similar, indicating that figure content and border ownership, but not border shape, determined performance. In preliminary results (N=1), we find that this asymmetry develops slowly; it is largely absent for a presentation time of 125 ms but close to maximal at 250 ms. In sum, figure-ground separation uses orientation differences in a way that depends on figure-ground assignment, suggesting a recurrent process that evolves over time.

Border ownership selectivity in area V4 occurs first in infragranular layers

Border ownership selectivity in area V4 occurs first in infragranular layers

Early recurrence enables figure border ownership

Rubin’s face-vase illusion demonstrates how one can switch back and forth between two different interpretations depending on how the figure outlines are assigned. In the primate visual system, assigning ownership along figure borders is encoded by neurons called the border ownership (BO) cells. Studies show that the responses of these neurons not only depend on the local features within their receptive fields, but also on contextual information. Despite two decades of studies on BO neurons, the ownership assignment mechanism in the brain is still unknown. Here, we propose a hierarchical recurrent model grounded on the hypothesis that neurons in the dorsal stream provide the context required for ownership assignment. Our proposed model incorporates early recurrence from the dorsal pathway as well as lateral modulations within the ventral stream. While dorsal modulations initiate the response difference to figure on either side of the border, lateral modulations enhance the difference. We found responses of our dorsally-modulated BO cells, similar to their biological counterparts, are invariant to size, position and solid/outlined figures. Moreover, our model BO cells exhibit comparable levels of reliability in the ownership signal to biological BO neurons. We found dorsal modulations result in high levels of accuracy and robustness for BO assignments in complex scenes compared to previous models based on ventral feedback. Finally, our experiments with illusory contours suggest that BO encoding could explain the perception of such contours in higher processing stages in the brain.

Analysis of spiking synchrony in visual cortex reveals distinct types of top-down modulation signals for spatial and object-based attention.

The activity of a border ownership selective (BOS) neuron indicates where a foreground object is located relative to its (classical) receptive field (RF). A population of BOS neurons thus provides an important component of perceptual grouping, the organization of the visual scene into objects. In previous theoretical work, it has been suggested that this grouping mechanism is implemented by a population of dedicated grouping (“G”) cells that integrate the activity of the distributed feature cells representing an object and, by feedback, modulate the same cells, thus making them border ownership selective. The feedback modulation by G cells is thought to also provide the mechanism for object-based attention. A recent modeling study showed that modulatory common feedback, implemented by synapses with N-methyl-D-aspartate (NMDA)-type glutamate receptors, accounts for the experimentally observed synchrony in spike trains of BOS neurons and the shape of cross-correlations between them, including its dependence on the attentional state. However, that study was limited to pairs of BOS neurons with consistent border ownership preferences, defined as two neurons tuned to respond to the same visual object, in which attention decreases synchrony. But attention has also been shown to increase synchrony in neurons with inconsistent border ownership selectivity. Here we extend the computational model from the previous study to fully understand these effects of attention. We postulate the existence of a second type of G-cell that represents spatial attention by modulating the activity of all BOS cells in a spatially defined area. Simulations of this model show that a combination of spatial and object-based mechanisms fully accounts for the observed pattern of synchrony between BOS neurons. Our results suggest that modulatory feedback from G-cells may underlie both spatial and object-based attention.

Learning Deep Conditional Neural Network for Image Segmentation

Combining Convolutional Neural Networks (CNNs) with Conditional Random Fields (CRFs) achieves great success among recent object segmentation methods. There are two advantages by such usage. First, CNNs can extract low-level features, which are very similar to the extracted features in primates’ primary visual cortex (V1). Second, CRFs can set up the relationship between input features and output labels in a direct way. In this paper, we extend the first advantage by using CNNs for low-level feature extraction and a Structured Random Forest (SRF)-based border ownership detector for high-level feature extraction, which are similar to the outputs of primates secondary visual cortex (V2). Compared to the CRF model, an improved Conditional Boltzmann Machine (CBM), which has a multi-channel visible layer, is proposed to model the relationship between predicted labels, local and global contexts of objects with multi-scale and multilevel features. Besides, our proposed CBM model is extended for object parsing by using multivisible branches instead of a single visible layer of CBM, which cannot only segment the whole body but also the parts of the body under. These visible branches use each branch for the segmentation of the whole body or one of the body parts. All branches share the same hidden layers of CBM and train the branches under an iterative way. By exploiting object parsing, the whole body segmentation performance of object is improved. To refine the segmentation output, two kinds of optimization algorithms are proposed. The superpixel-based algorithm can re-label the overlapped regions of multiple kinds of objects. The other curve correction algorithm corrects the edges of segmented object parts by using smooth edges under a curve similarity criterion. Experiments demonstrate that our models yield competitive results for object segmentation on the PASCAL VOC 2012 dataset and for object parsing on the PennFudan Pedestrian Parsing dataset, Pedestrian Parsing Surveillance Scenes dataset, Horse-Cow parsing dataset, and PASCAL Quadrupeds dataset.

Figure-Ground Organization in Natural Scenes: Performance of a Recurrent Neural Model Compared with Neurons of Area V2.

A crucial step in understanding visual input is its organization into meaningful components, in particular object contours and partially occluded background structures. This requires that all contours are assigned to either the foreground or the background (border ownership assignment). While earlier studies showed that neurons in primate extrastriate cortex signal border ownership for simple geometric shapes, recent studies show consistent border ownership coding also for complex natural scenes. In order to understand how the brain performs this task, we developed a biologically plausible recurrent neural network that is fully image computable. Our model uses local edge detector () cells and grouping () cells whose activity represents proto-objects based on the integration of local feature information. cells send modulatory feedback connections to those cells that caused their activation, making the cells border ownership selective. We found close agreement between our model and neurophysiological results in terms of the timing of border ownership signals (BOSs) as well as the consistency of BOSs across scenes. We also benchmarked our model on the Berkeley Segmentation Dataset and achieved performance comparable to recent state-of-the-art computer vision approaches. Our proposed model provides insight into the cortical mechanisms of figure-ground organization.

Border ownership-dependent tilt aftereffect for shape defined by binocular disparity and motion parallax.

Discerning objects from their surrounds (i.e., figure-ground segmentation) in a way that guides adaptive behaviors is a fundamental task of the brain. Neurophysiological work has revealed a class of cells in the macaque visual cortex that may be ideally suited to support this neural computation: border ownership cells (Zhou H, Friedman HS, von der Heydt R. J Neurosci 20: 6594–6611, 2000). These orientation-tuned cells appear to respond conditionally to the borders of objects. A behavioral correlate supporting the existence of these cells in humans was demonstrated with two-dimensional luminance-defined objects (von der Heydt R, Macuda T, Qiu FT. J Opt Soc Am A Opt Image Sci Vis 22: 2222–2229, 2005). However, objects in our natural visual environments are often signaled by complex cues, such as motion and binocular disparity. Thus for border ownership systems to effectively support figure-ground segmentation and object depth ordering, they must have access to information from multiple depth cues with strict depth order selectivity. Here we measured in humans (of both sexes) border ownership-dependent tilt aftereffects after adaptation to figures defined by either motion parallax or binocular disparity. We find that both depth cues produce a tilt aftereffect that is selective for figure-ground depth order. Furthermore, we find that the effects of adaptation are transferable between cues, suggesting that these systems may combine depth cues to reduce uncertainty (Bülthoff HH, Mallot HA. J Opt Soc Am A 5: 1749–1758, 1988). These results suggest that border ownership mechanisms have strict depth order selectivity and access to multiple depth cues that are jointly encoded, providing compelling psychophysical support for their role in figure-ground segmentation in natural visual environments.NEW & NOTEWORTHY Figure-ground segmentation is a critical function that may be supported by “border ownership” neural systems that conditionally respond to object borders. We measured border ownership-dependent tilt aftereffects to figures defined by motion parallax or binocular disparity and found aftereffects for both cues. These effects were transferable between cues but selective for figure-ground depth order, suggesting that the neural systems supporting figure-ground segmentation have strict depth order selectivity and access to multiple depth cues that are jointly encoded.

Predicting Illusory Contours Without Extracting Special Image Features.

Boundary completion is one of the desired properties of a robust object boundary detection model, since in real-word images the object boundaries are commonly not fully and clearly seen. An extreme example of boundary completion occurs in images with illusory contours, where the visual system completes boundaries in locations without intensity gradient. Most illusory contour models extract special image features, such as L and T junctions, while the task is known to be a difficult issue in real-world images. The proposed model uses a functional optimization approach, in which a cost value is assigned to any boundary arrangement to find the arrangement with minimal cost. The functional accounts for basic object properties, such as alignment with the image, object boundary continuity, and boundary simplicity. The encoding of these properties in the functional does not require special features extraction, since the alignment with the image only requires extraction of the image edges. The boundary arrangement is represented by a border ownership map, holding object boundary segments in discrete locations and directions. The model finds multiple possible image interpretations, which are ranked according to the probability that they are supposed to be perceived. This is achieved by using a novel approach to represent the different image interpretations by multiple functional local minima. The model is successfully applied to objects with real and illusory contours. In the case of Kanizsa illusion the model predicts both illusory and real (pacman) image interpretations. The model is a proof of concept and is currently restricted to synthetic gray-scale images with solid regions.

Learning Invariant Features in Modulatory Networks through Conflict and Ambiguity.

This work lays the foundation for a framework of cortical learning based on the idea of a competitive column, which is inspired by the functional organization of neurons in the cortex. A column describes a prototypical organization for neurons that gives rise to an ability to learn scale, rotation, and translation-invariant features. This is empowered by a recently developed learning rule, conflict learning, which enables the network to learn over both driving and modulatory feedforward, feedback, and lateral inputs. The framework is further supported by introducing both a notion of neural ambiguity and an adaptive threshold scheme. Ambiguity, which captures the idea that too many decisions lead to indecision, gives the network a dynamic way to resolve locally ambiguous decisions. The adaptive threshold operates over multiple timescales to regulate neural activity under the varied arrival timings of input in a highly interconnected multilayer network with feedforward and feedback. The competitive column architecture is demonstrated on a large-scale (54,000 neurons and 18 million synapses), invariant model of border ownership. The model is trained on four simple, fixed-scale shapes: two squares, one rectangle, and one symmetric L-shape. Tested on 1899 synthetic shapes of varying scale and complexity, the model correctly assigned border ownership with 74% accuracy. The model's abilities were also illustrated on contours of objects taken from natural images. Combined with conflict learning, the competitive column and ambiguity give a better intuitive understanding of how feedback, modulation, and inhibition may interact in the brain to influence activation and learning.

Saliency model based on a neural population for integrating figure direction and organizing Border Ownership

Attentional selection is a function of the brain that allocates computational resources momentarily to the most important part of a visual scene. Saliency map models have been used to predict the location of attentional selection and gaze. Border Ownership (BO) indicates the direction of the figure with respect to the border. I here propose a biologically plausible saliency model based on neural population for integrating the activities of intermediate-level visual areas with neurons selective to BO. A variety of BO organizations produces a population of model neurons that represent the grouping structure. In the model I propose, the interactions and the population responses of these model neurons underlie the determination of saliency and the accurate prediction of gaze location. I tested 100 patterns for BO organizations and found that the proposed saliency model not only reproduced the characteristics of perceptual organization but also captured object locations in natural images. Furthermore, the saliency model based on the population responses of the BO organization significantly improved the gaze prediction accuracy compared with previous saliency-based models. These results suggest a crucial role for a wide variety of BO organizations and neural population coding to determine saliency mediating attentional selection and to predict gaze location.

Interactions Elicited by the Contradiction Between Figure Direction Discrimination and Figure-Ground Segregation.

Figure-ground (FG) segregation that separates an object from the rest of the image is a fundamental problem in vision science. A majority of neurons in monkey V2 showed the selectivity to border ownership (BO) that indicates which side of a contour owns the border. Although BO could be a precursor of FG segregation, the contribution of BO to FG segregation has not been clarified. Because FG segregation is the perception of the global region that belongs to an object, whereas BO determination provides the local direction of figure (DOF) along a contour, a spatial integration of BO might be expected for the generation of FG. To understand the mechanisms underlying the perception of figural regions, we investigated the interaction between the local BO determination and the global FG segregation through the quantitative analysis of the visual perception and the spatiotemporal characteristics of eye movements. We generated a set of novel stimuli in which translucency induces local DOF along the contour and global FG independently so that DOF and FG could be either consistent or contradictory. The perceptual responses showed better performance in DOF discrimination than FG segregation, supporting distinct mechanisms for the DOF discrimination and the FG segregation. We examined whether the contradiction between DOF and FG modulates the eye movement while participants judged DOF and FG. The duration of the first eye fixation was modulated by the contradiction during FG segregation but not DOF discrimination, suggesting a sequential processing from the BO determination to the FG segregation. These results of human perception and eye fixation provide important clues for understanding the visual processing for FG segregation.

Border Ownership Assignment based on Dorsal and Horizontal Modulations

Border Ownership Assignment based on Dorsal and Horizontal Modulations