Shape-selective Neurons Research Articles

Modeling diverse responses to filled and outline shapes in macaque V4.

Visual area V4 is an important midlevel cortical processing stage that subserves object recognition in primates. Studies investigating shape coding in V4 have largely probed neuronal responses with filled shapes, i.e., shapes defined by both a boundary and an interior fill. As a result, we do not know whether form-selective V4 responses are dictated by boundary features alone or if interior fill is also important. We studied 43 V4 neurons in two male macaque monkeys ( Macaca mulatta) with a set of 362 filled shapes and their corresponding outlines to determine how interior fill modulates neuronal responses in shape-selective neurons. Only a minority of neurons exhibited similar response strength and shape preferences for filled and outline stimuli. A majority responded preferentially to one stimulus category (either filled or outline shapes) and poorly to the other. Our findings are inconsistent with predictions of the hierarchical-max (HMax) V4 model that builds form selectivity from oriented boundary features and takes little account of attributes related to object surface, such as the phase of the boundary edge. We modified the V4 HMax model to include sensitivity to interior fill by either removing phase-pooling or introducing unoriented units at the V1 level; both modifications better explained our data without increasing the number of free parameters. Overall, our results suggest that boundary orientation and interior surface information are both maintained until at least the midlevel visual representation, consistent with the idea that object fill is important for recognition and perception in natural vision. NEW & NOTEWORTHY The shape of an object's boundary is critical for identification; consistent with this idea, models of object recognition predict that filled and outline versions of a shape are encoded similarly. We report that many neurons in a midlevel visual cortical area respond differently to filled and outline shapes and modify a biologically plausible model to account for our data. Our results suggest that representations of boundary shape and surface fill are interrelated in visual cortex.

27th Annual Computational Neuroscience Meeting (CNS*2018): Part One

P170 Neural model for the recognition of agency and social interaction from abstract stimuliMohammad Hovaidi Ardestani1, Martin Giese2, Nitin Saini2 1University Clinic Tübingen, Tübingen, Germany; 2Center for Integrative Neuroscience & University Clinic Tübingen, Dept of Cogniitive Neurology, Germany Correspondence: Martin Giese (martin.giese@uni-tuebingen.de)BMC Neuroscience 2018, 19(Suppl 2):P170Introduction: Humans derive spontaneously judgements about agency and social interactions from strongly impoverished stimuli, as impressively demonstrated by the seminal work by Heider and Simmel (1944). The neural circuits that derive such judgements from image sequences are entirely unknown. It has been hypothesized that this visual function is based on high-level cognitive processes, such as probabilistic reasoning. Taking an alternative approach, we show that such functions can be accomplished by relatively elementary neural networks that can be implemented by simple physiologically plausible neural mechanisms, exploiting an appropriately structure hierarchical (deep) neural model of the visual pathway.Methods: Extending classical biologically-inspired models for object and action perception (Riesenhuber & Poggio 1999; Giese & Poggio 2003) by a front-end that exploits deep learning for the construction of low and mid-level feature detectors, we built a hierarchical neural model that reproduces elementary psychophysical results on animacy and social perception from abstract stimuli. The lower hierarchy levels of the model consist of position-variant neural feature detectors that extract orientation and intermediately complex shape features. The next-higher level is formed by shape-selective neurons that are not completely position-invariant, which extract the 2D positions and orientation of moving agents. A second pathway extracts the 2D motion of the moving agents. Exploiting a gain-field network, we compute the relative positions of the moving agents. The top layers of the model combine the mentioned features into more complex high-level features that represent the speed, smoothness of motion and spatial relationships of the moving agents. The highest level of the model consists of neurons that have learned to classify the agency of the motions, and different categories of social interactions.Results: Based on input video sequences, the model successfully reproduces results of Tremoulet and Feldman (2000) on the dependence of perceived animacy on motion parameters, and its dependence on the alignment of motion and body axis (Hernik et al. 2013). In addition, the model correctly classifies four categories of social interactions that have been frequently tested in the psychophysical literature (following, chasing, fighting, guarding) (e.g. Scholl and McCarthy, 2012; McAleer et al., 2011a).Conclusion: Using simple physiologically plausible neural circuits, the model accounts simultaneously for a variety of effects related to animacy and social interaction perception. This leads to interesting predictions about neurons involved in the visual processing of such stimuli. Acknowledgement This work was supported by: HFSP RGP0036/2016; the European Commission HBP FP7-ICT2013-FET-F/604102 and COGIMON H2020-644727, and the DFG GZ: GI 305/4-1 and KA 1258/15-1.

Joint coding of shape and blur in area V4

Edge blur, a prevalent feature of natural images, is believed to facilitate multiple visual processes including segmentation and depth perception. Furthermore, image descriptions that explicitly combine blur and shape information provide complete representations of naturalistic scenes. Here we report the first demonstration of blur encoding in primate visual cortex: neurons in macaque V4 exhibit tuning for both object shape and boundary blur, with observed blur tuning not explained by potential confounds including stimulus size, intensity, or curvature. A descriptive model wherein blur selectivity is cast as a distinct neural process that modulates the gain of shape-selective V4 neurons explains observed data, supporting the hypothesis that shape and blur are fundamental features of a sufficient neural code for natural image representation in V4.

Inhibition-augmented COSFIRE model of shape-selective neurons

Inhibition is a phenomenon that occurs in different areas of the brain, including the visual cortex. For instance, the responses of some shape-selective neurons in the inferotemporal cortex are suppressed by the presence of certain shape contour parts in their receptive fields. This suppression phenomenon is thought to increase the selectivity of such neurons. We propose an inhibition-augmented model of shape-selective neurons, as an advancement of the trainable filter approach called combination of shifted filter responses (COSFIRE). We use a positive prototype pattern and a set of negative prototype patterns to automatically configure an inhibition-augmented model. The configuration involves the selection of responses of a bank of Gabor filters (models of V1/V2 neurons) that provide excitatory or inhibitory input(s). We compute the output of the model as the excitatory input minus a fraction of the maximum of the inhibitory inputs. The configured model responds to patterns that are similar to the positive prototype but does not respond to patterns similar to the negative prototype(s). We demonstrate the effectiveness of the proposed model in shape recognition. We use the Graphics Recognition (GREC2011) benchmark dataset and demonstrate that the proposed inhibition-augmented modeling technique increases selectivity of the COSFIRE model.

Single Canonical Model of Reflexive Memory and Spatial Attention.

Many neurons in the dorsal and ventral visual stream have the property that after a brief visual stimulus presentation in their receptive field, the spiking activity in these neurons persists above their baseline levels for several seconds. This maintained activity is not always correlated with the monkey’s task and its origin is unknown. We have previously proposed a simple neural network model, based on shape selective neurons in monkey lateral intraparietal cortex, which predicts the valence and time course of reflexive (bottom-up) spatial attention. In the same simple model, we demonstrate here that passive maintained activity or short-term memory of specific visual events can result without need for an external or top-down modulatory signal. Mutual inhibition and neuronal adaptation play distinct roles in reflexive attention and memory. This modest 4-cell model provides the first simple and unified physiologically plausible mechanism of reflexive spatial attention and passive short-term memory processes.

A Growth-Cone Model for the Spread of Object-Based Attention during Contour Grouping

Object-based attention can group image elements of spatially extended objects into coherent representations, but its mechanisms have remained unclear. The mechanisms for object-based attention may include shape-selective neurons in higher visual cortical areas that feed back to lower areas to simultaneously enhance the representation of all image elements of a relevant shape. Another, nonexclusive mechanism is the spread of attention in early visual cortex according to Gestalt rules, which could successively add new elements to a growing object representation. We investigated the dynamics of object-based attention in the primary visual cortex of monkeys trained to perform a contour-grouping task. The animals mentally traced a target curve through the visual field and ignored a distracting curve. Neuronal responses elicited by the target curve were enhanced relative to those elicited by distracting curve. Remarkably, the response enhancement was delayed for neurons with receptive fields farther from the start of the tracing process. We could therefore measure propagation speed and found that it was low if curves were nearby and that it increased if curves were far apart. The results are well explained by an "attentional growth-cone" model, which holds that the response enhancement can spread in multiple visual cortical areas with different receptive field sizes at a speed of approximately 50 ms per receptive field. Our findings support an active role for early visual areas in object-based attention because neurons in these areas gradually spread enhanced activity over the representation of relevant objects.

The role of visual area V4 in the discrimination of partially occluded shapes.

The primate brain successfully recognizes objects, even when they are partially occluded. To begin to elucidate the neural substrates of this perceptual capacity, we measured the responses of shape-selective neurons in visual area V4 while monkeys discriminated pairs of shapes under varying degrees of occlusion. We found that neuronal shape selectivity always decreased with increasing occlusion level, with some neurons being notably more robust to occlusion than others. The responses of neurons that maintained their selectivity across a wider range of occlusion levels were often sufficiently sensitive to support behavioral performance. Many of these same neurons were distinctively selective for the curvature of local boundary features and their shape tuning was well fit by a model of boundary curvature (curvature-tuned neurons). A significant subset of V4 neurons also signaled the animal's upcoming behavioral choices; these decision signals had short onset latencies that emerged progressively later for higher occlusion levels. The time course of the decision signals in V4 paralleled that of shape selectivity in curvature-tuned neurons: shape selectivity in curvature-tuned neurons, but not others, emerged earlier than the decision signals. These findings provide evidence for the involvement of contour-based mechanisms in the segmentation and recognition of partially occluded objects, consistent with psychophysical theory. Furthermore, they suggest that area V4 participates in the representation of the relevant sensory signals and the generation of decision signals underlying discrimination.

Modulation of neuronal activity with cue-invariant shape discrimination in the primate superior temporal sulcus

Shape perception can be achieved based on various cues such as luminance, color, texture, depth and motion. To investigate common neural mechanisms underlying shape perception cued by various visual attributes, we examined single-neuron activity in the monkey anterior superior temporal sulcus (STS) in response to shapes defined by luminance and motion cues during shape discrimination. We found cortical mapping with respect to selectivity for shapes as well as for direction of motion in the STS. About 90% of shape-selective neurons were located in the lower bank of STS (lSTS) assigned to the ventral pathway, while about 80% of direction-selective neurons existed in the upper bank of STS (uSTS) assigned to the dorsal pathway. The neurons showing selectivity for both shape and motion coexisted in lSTS as well as uSTS. This result indicates that integration or convergence of shape information and motion information can occur in both banks of STS. About 90% of STS neurons showing selectivity both for shapes defined by luminance cue and for shapes defined by motion cue were located in lSTS. They showed a highly similar shape preference between the different visual attributes, indicating cue-invariant shape selectivity. The cue-invariant shape-selectivity was modulated with target selection as well as with discrimination performance of monkeys. These results suggest that lSTS could be involved in cue-invariant shape discrimination, but not the uSTS.

Time Course of Shape and Category Selectivity Revealed by EEG Rapid Adaptation

A hallmark of human cognition is the ability to rapidly assign meaning to sensory stimuli. It has been suggested that this fast visual object categorization ability is accomplished by a feedforward processing hierarchy consisting of shape-selective neurons in occipito-temporal cortex that feed into task circuits in frontal cortex computing conceptual category membership. We performed an EEG rapid adaptation study to test this hypothesis. Participants were trained to categorize novel stimuli generated with a morphing system that precisely controlled both stimulus shape and category membership. We subsequently performed EEG recordings while participants performed a category matching task on pairs of successively presented stimuli. We used space-time cluster analysis to identify channels and latencies exhibiting selective neural responses. Neural signals before 200 msec on posterior channels demonstrated a release from adaptation for shape changes, irrespective of category membership, compatible with a shape- but not explicitly category-selective neural representation. A subsequent cluster with anterior topography appeared after 200 msec and exhibited release from adaptation consistent with explicit categorization. These signals were subsequently modulated by perceptual uncertainty starting around 300 msec. The degree of category selectivity of the anterior signals was strongly predictive of behavioral performance. We also observed a posterior category-selective signal after 300 msec exhibiting significant functional connectivity with the initial anterior category-selective signal. In summary, our study supports the proposition that perceptual categorization is accomplished by the brain within a quarter second through a largely feedforward process culminating in frontal areas, followed by later category-selective signals in posterior regions.

Three-dimensional Shape Coding in Grasping Circuits: A Comparison between the Anterior Intraparietal Area and Ventral Premotor Area F5a

Depth information is necessary for adjusting the hand to the three-dimensional (3-D) shape of an object to grasp it. The transformation of visual information into appropriate distal motor commands is critically dependent on the anterior intraparietal area (AIP) and the ventral premotor cortex (area F5), particularly the F5p sector. Recent studies have demonstrated that both AIP and the F5a sector of the ventral premotor cortex contain neurons that respond selectively to disparity-defined 3-D shape. To investigate the neural coding of 3-D shape and the behavioral role of 3-D shape-selective neurons in these two areas, we recorded single-cell activity in AIP and F5a during passive fixation of curved surfaces and during grasping of real-world objects. Similar to those in AIP, F5a neurons were either first- or second-order disparity selective, frequently showed selectivity for discrete approximations of smoothly curved surfaces that contained disparity discontinuities, and exhibited mostly monotonic tuning for the degree of disparity variation. Furthermore, in both areas, 3-D shape-selective neurons were colocalized with neurons that were active during grasping of real-world objects. Thus, area AIP and F5a contain highly similar representations of 3-D shape, which is consistent with the proposed transfer of object information from AIP to the motor system through the ventral premotor cortex.

Trainable COSFIRE Filters for Keypoint Detection and Pattern Recognition

Keypoint detection is important for many computer vision applications. Existing methods suffer from insufficient selectivity regarding the shape properties of features and are vulnerable to contrast variations and to the presence of noise or texture. We propose a trainable filter which we call Combination Of Shifted FIlter REsponses (COSFIRE) and use for keypoint detection and pattern recognition. It is automatically configured to be selective for a local contour pattern specified by an example. The configuration comprises selecting given channels of a bank of Gabor filters and determining certain blur and shift parameters. A COSFIRE filter response is computed as the weighted geometric mean of the blurred and shifted responses of the selected Gabor filters. It shares similar properties with some shape-selective neurons in visual cortex, which provided inspiration for this work. We demonstrate the effectiveness of the proposed filters in three applications: the detection of retinal vascular bifurcations (DRIVE dataset: 98.50 percent recall, 96.09 percent precision), the recognition of handwritten digits (MNIST dataset: 99.48 percent correct classification), and the detection and recognition of traffic signs in complex scenes (100 percent recall and precision). The proposed COSFIRE filters are conceptually simple and easy to implement. They are versatile keypoint detectors and are highly effective in practical computer vision applications.

The roles of endstopped and curvature tuned computations in a hierarchical representation of 2D shape.

That shape is important for perception has been known for almost a thousand years (thanks to Alhazen in 1083) and has been a subject of study ever since by scientists and phylosophers (such as Descartes, Helmholtz or the Gestalt psychologists). Shapes are important object descriptors. If there was any remote doubt regarding the importance of shape, recent experiments have shown that intermediate areas of primate visual cortex such as V2, V4 and TEO are involved in analyzing shape features such as corners and curvatures. The primate brain appears to perform a wide variety of complex tasks by means of simple operations. These operations are applied across several layers of neurons, representing increasingly complex, abstract intermediate processing stages. Recently, new models have attempted to emulate the human visual system. However, the role of intermediate representations in the visual cortex and their importance have not been adequately studied in computational modeling.This paper proposes a model of shape-selective neurons whose shape-selectivity is achieved through intermediate layers of visual representation not previously fully explored. We hypothesize that hypercomplex - also known as endstopped - neurons play a critical role to achieve shape selectivity and show how shape-selective neurons may be modeled by integrating endstopping and curvature computations. This model - a representational and computational system for the detection of 2-dimensional object silhouettes that we term 2DSIL - provides a highly accurate fit with neural data and replicates responses from neurons in area V4 with an average of 83% accuracy. We successfully test a biologically plausible hypothesis on how to connect early representations based on Gabor or Difference of Gaussian filters and later representations closer to object categories without the need of a learning phase as in most recent models.

Shape encoding consistency across colors in primate V4

Neurons in primate cortical area V4 are sensitive to the form and color of visual stimuli. To determine whether form selectivity remains consistent across colors, we studied the responses of single V4 neurons in awake monkeys to a set of two-dimensional shapes presented in two different colors. For each neuron, we chose two colors that were visually distinct and that evoked reliable and different responses. Across neurons, the correlation coefficient between responses in the two colors ranged from -0.03 to 0.93 (median 0.54). Neurons with highly consistent shape responses, i.e., high correlation coefficients, showed greater dispersion in their responses to the different shapes, i.e., greater shape selectivity, and also tended to have less eccentric receptive field locations; among shape-selective neurons, shape consistency ranged from 0.16 to 0.93 (median 0.63). Consistency of shape responses was independent of the physical difference between the stimulus colors used and the strength of neuronal color tuning. Finally, we found that our measurement of shape response consistency was strongly influenced by the number of stimulus repeats: consistency estimates based on fewer than 10 repeats were substantially underestimated. In conclusion, our results suggest that neurons that are likely to contribute to shape perception and discrimination exhibit shape responses that are largely consistent across colors, facilitating the use of simpler algorithms for decoding shape information from V4 neuronal populations.

Inferotemporal Cortex Subserves Three-Dimensional Structure Categorization

We perceive real-world objects as three-dimensional (3D), yet it is unknown which brain area underlies our ability to perceive objects in this way. The macaque inferotemporal (IT) cortex contains neurons that respond selectively to 3D structures defined by binocular disparity. To examine the causal role of IT in the categorization of 3D structures, we electrically stimulated clusters of IT neurons with a similar 3D-structure preference while monkeys performed a 3D-structure categorization task. Microstimulation of 3D-structure-selective IT clusters caused monkeys to choose the preferred structure of the 3D-structure-selective neurons considerably more often. Microstimulation in IT also accelerated the monkeys' choice for the preferred structure, while delaying choices corresponding to the nonpreferred structure of a given site. These findings reveal that 3D-structure-selective neurons in IT contribute to the categorization of 3D objects.

Three-dimensional shape from second-order orientation flows

In images of textured surfaces, orientation flows formed by perspective convergence invariably convey 3D shape. We show that orientation flows formed by contrast-modulated (CM) and illusory contours (IC) convey 3D shape, and that both stimulus types induce 3D shape aftereffects on CM and IC test stimuli. Adaptation to luminance-modulated (LM) orientation flows induce robust 3D shape aftereffects on CM and IC tests, however, aftereffects using CM/IC adapting stimuli on LM tests were substantially weaker. These results can be explained by the adaptation of 3D shape-selective neurons that invariantly extract first- and second-order orientation flows from striate and extra-striate signals, which receive stronger input from neurons selective for first-order orientation flows.

The Relationship between Saccadic Suppression and Perceptual Stability

Introspection makes it clear that we do not see the visual motion generated by our saccadic eye movements. We refer to the lack of awareness of the motion across the retina that is generated by a saccade as saccadic omission [1]: the visual stimulus generated by the saccade is omitted from our subjective awareness. In the laboratory, saccadic omission is often studied by investigating saccadic suppression, the reduction in visual sensitivity before and during a saccade (see Ross et al. [2] and Wurtz [3] for reviews). We investigated whether perceptual stability requires that a mechanism like saccadic suppression removes perisaccadic stimuli from visual processing to prevent their presumed harmful effect on perceptual stability [4, 5]. Our results show that a stimulus that undergoes saccadic omission can nevertheless generate a shape contrast illusion. This illusion can be generated when the inducer and test stimulus are separated in space and is therefore thought to be generated at a later stage of visual processing [6]. This shows that perceptual stability is attained without removing stimuli from processing and suggests a conceptually new view of perceptual stability in which perisaccadic stimuli are processed by the early visual system, but these signals are prevented from reaching awareness at a later stage of processing.

Three-dimensional shape from second-order orientation flows

In images of textured surfaces, orientation flows formed by perspective convergence invariably convey 3D shape. We show that orientation flows formed by contrast-modulated (CM) and illusory contours (IC) convey 3D shape, and that both stimulus types induce 3D shape aftereffects on CM and IC test stimuli. Adaptation to luminance-modulated (LM) orientation flows induce robust 3D shape aftereffects on CM and IC tests, however, aftereffects using CM/IC adapting stimuli on LM tests were substantially weaker. These results can be explained by the adaptation of 3D shape-selective neurons that invariantly extract first- and second-order orientation flows from striate and extra-striate signals, which receive stronger input from neurons selective for first-order orientation flows.

Processing of Kinetic Boundaries in Macaque V4

We used gratings and shapes defined by relative motion to study selectivity for static kinetic boundaries in macaque V4 neurons. Kinetic gratings were generated by random pixels moving in opposite directions in the neighboring bars, either parallel to the orientation of the boundary (parallel kinetic grating) or perpendicular to the boundary (orthogonal kinetic grating). Neurons were also tested with static, luminance defined gratings to establish cue invariance. In addition, we used eight shapes defined either by relative motion or by luminance contrast, as used previously to test cue invariance in the infero-temporal (IT) cortex. A sizeable fraction (10-20%) of the V4 neurons responded selectively to kinetic patterns. Most neurons selective for kinetic contours had receptive fields (RFs) within the central 10 degrees of the visual field. Neurons selective for the orientation of kinetic gratings were defined as having similar orientation preferences for the two types of kinetic gratings, and the vast majority of these neurons also retained the same orientation preference for luminance defined gratings. Also, kinetic shape selective neurons had similar shape preferences when the shape was defined by relative motion or by luminance contrast, showing a cue-invariant form processing in V4. Although shape selectivity was weaker in V4 than what has been reported in the IT cortex, cue invariance was similar in the two areas, suggesting that invariance for luminance and motion cues of IT originates in V4. The neurons selective for kinetic patterns tended to be clustered within dorsal V4.

Neuronal activity in the primate prefrontal cortex in the process of motor selection based on two behavioral rules.

This study examined neuronal activity in the prefrontal cortex (PF) involved in the process of motor selection in accordance with two behavioral rules. We trained two monkeys to select a target based on the integration of memorized and current sensory information. Initially, a sample cue (triangle or circle) appeared at one of three locations (top, left, or right) for 1 s. After a 3-s delay, one of two types of choice cue appeared. The first type asked the monkeys to reach for a target by matching the location (location-matching task). The second type asked the monkeys to reach for a target by matching the shape (shape-matching task). The choice cue for location matching consisted of either three circles or three triangles, and the choice cue for shape matching consisted of a circle and a triangle. When the color of the choice cue changed from red to green 1.5 s later (GO signal), the monkeys touched the correct object to obtain a reward. We found cue-, delay-, choice-, and movement-related neuronal activity in the lateral prefrontal cortex. During the sample cue presentation and delay periods, we found selective neuronal activity for the location or shape of the sample cue. Shape-selective neurons were located more anteriorly in the ventral bank of the principal sulcus and inferior convexity area, whereas location-selective neurons were more posteriorly. After the choice cue appeared, we found three main types of neuronal activity in the critical period when the subject selected the future target: 1) activity reflecting past sensory information (the location or shape of the sample cue presented 3 s earlier), 2) activity selective for the configuration of the current choice cue, and 3) activity reflecting the properties (location or shape) of the future target. During the motor-response period, we found neuronal activity selective for the location or shape of the reaching target. When muscimol was microinjected into the ventral bank of principal sulcus and inferior convexity area, the performance of both tasks was impaired. Furthermore, we found that the wealth of neuronal activity in the PF that seemed to play a role in motor selection was rarely seen in the primary motor cortex.

Cortical correlate of pattern backward masking.

The perception of a briefly presented shape is strongly impaired when it is followed by another pattern, a phenomenon called backward masking. We found that the vast majority of a sample of shape-selective neurons in the macaque inferior temporal cortex respond selectively to backward-masked shapes, although these shapes could not be discriminated by human and monkey subjects. However, this selective response was brief, since it was either interrupted by the mask or overridden by a response to the mask itself. We show that reliable discrimination of briefly presented shapes by single neurons depends on the temporal integration of the response. Presentation of the mask, however, reduces the number of spikes available for integration, explaining backward masking. These results also provide direct neurophysiological evidence for the "interruption theory" of backward masking.