High-level Object Representations Research Articles

Learning cortical hierarchies with temporal Hebbian updates.

A key driver of mammalian intelligence is the ability to represent incoming sensory information across multiple abstraction levels. For example, in the visual ventral stream, incoming signals are first represented as low-level edge filters and then transformed into high-level object representations. Similar hierarchical structures routinely emerge in artificial neural networks (ANNs) trained for object recognition tasks, suggesting that similar structures may underlie biological neural networks. However, the classical ANN training algorithm, backpropagation, is considered biologically implausible, and thus alternative biologically plausible training methods have been developed such as Equilibrium Propagation, Deep Feedback Control, Supervised Predictive Coding, and Dendritic Error Backpropagation. Several of those models propose that local errors are calculated for each neuron by comparing apical and somatic activities. Notwithstanding, from a neuroscience perspective, it is not clear how a neuron could compare compartmental signals. Here, we propose a solution to this problem in that we let the apical feedback signal change the postsynaptic firing rate and combine this with a differential Hebbian update, a rate-based version of classical spiking time-dependent plasticity (STDP). We prove that weight updates of this form minimize two alternative loss functions that we prove to be equivalent to the error-based losses used in machine learning: the inference latency and the amount of top-down feedback necessary. Moreover, we show that the use of differential Hebbian updates works similarly well in other feedback-based deep learning frameworks such as Predictive Coding or Equilibrium Propagation. Finally, our work removes a key requirement of biologically plausible models for deep learning and proposes a learning mechanism that would explain how temporal Hebbian learning rules can implement supervised hierarchical learning.

Representations of Temporal Community Structure in Hippocampus and Precuneus Predict Inductive Reasoning Decisions.

Our understanding of the world is shaped by inferences about underlying structure. For example, at the gym, you might notice that the same people tend to arrive around the same time and infer that they are friends that work out together. Consistent with this idea, after participants are presented with a temporal sequence of objects that follows an underlying community structure, they are biased to infer that objects from the same community share the same properties. Here, we used fMRI to measure neural representations of objects after temporal community structure learning and examine how these representations support inference about object relationships. We found that community structure learning affected inferred object similarity: When asked to spatially group items based on their experience, participants tended to group together objects from the same community. Neural representations in perirhinal cortex predicted individual differences in object grouping, suggesting that high-level object representations are affected by temporal community learning. Furthermore, participants were biased to infer that objects from the same community would share the same properties. Using computational modeling of temporal learning and inference decisions, we found that inductive reasoning is influenced by both detailed knowledge of temporal statistics and abstract knowledge of the temporal communities. The fidelity of temporal community representations in hippocampus and precuneus predicted the degree to which temporal community membership biased reasoning decisions. Our results suggest that temporal knowledge is represented at multiple levels of abstraction, and that perirhinal cortex, hippocampus, and precuneus may support inference based on this knowledge.

Training for object recognition with increasing spatial frequency: A comparison of deep learning with human vision.

The ontogenetic development of human vision and the real-time neural processing of visual input exhibit a striking similarity—a sensitivity toward spatial frequencies that progresses in a coarse-to-fine manner. During early human development, sensitivity for higher spatial frequencies increases with age. In adulthood, when humans receive new visual input, low spatial frequencies are typically processed first before subsequent processing of higher spatial frequencies. We investigated to what extent this coarse-to-fine progression might impact visual representations in artificial vision and compared this to adult human representations. We simulated the coarse-to-fine progression of image processing in deep convolutional neural networks (CNNs) by gradually increasing spatial frequency information during training. We compared CNN performance after standard and coarse-to-fine training with a wide range of datasets from behavioral and neuroimaging experiments. In contrast to humans, CNNs that are trained using the standard protocol are very insensitive to low spatial frequency information, showing very poor performance in being able to classify such object images. By training CNNs using our coarse-to-fine method, we improved the classification accuracy of CNNs from 0% to 32% on low-pass-filtered images taken from the ImageNet dataset. The coarse-to-fine training also made the CNNs more sensitive to low spatial frequencies in hybrid images with conflicting information in different frequency bands. When comparing differently trained networks on images containing full spatial frequency information, we saw no representational differences. Overall, this integration of computational, neural, and behavioral findings shows the relevance of the exposure to and processing of inputs with variation in spatial frequency content for some aspects of high-level object representations.

Prior object-knowledge sharpens properties of early visual feature-detectors

Early stages of visual processing are carried out by neural circuits activated by simple and specific features, such as the orientation of an edge. A fundamental question in human vision is how the brain organises such intrinsically local information into meaningful properties of objects. Classic models of visual processing emphasise a one-directional flow of information from early feature-detectors to higher-level information-processing. By contrast to this view, and in line with predictive-coding models of perception, here, we provide evidence from human vision that high-level object representations dynamically interact with the earliest stages of cortical visual processing. In two experiments, we used ambiguous stimuli that, depending on the observer’s prior object-knowledge, can be perceived as either coherent objects or as a collection of meaningless patches. By manipulating object knowledge we were able to determine its impact on processing of low-level features while keeping sensory stimulation identical. Both studies demonstrate that perception of local features is facilitated in a manner consistent with an observer’s high-level object representation (i.e., with no effect on object-inconsistent features). Our results cannot be ascribed to attentional influences. Rather, they suggest that high-level object representations interact with and sharpen early feature-detectors, optimising their performance for the current perceptual context.

Multisensory Part-based Representations of Objects in Human Lateral Occipital Cortex.

The format of high-level object representations in temporal-occipital cortex is a fundamental and as yet unresolved issue. Here we use fMRI to show that human lateral occipital cortex (LOC) encodes novel 3-D objects in a multisensory and part-based format. We show that visual and haptic exploration of objects leads to similar patterns of neural activity in human LOC and that the shared variance between visually and haptically induced patterns of BOLD contrast in LOC reflects the part structure of the objects. We also show that linear classifiers trained on neural data from LOC on a subset of the objects successfully predict a novel object based on its component part structure. These data demonstrate a multisensory code for object representations in LOC that specifies the part structure of objects.

Multiple Level Crowding: Crowding at the Object Parts Level and at the Object Configural level.

In crowding, identification of a peripheral target in the presence of nearby flankers is worse than when the target appears alone. Prevailing theories hold that crowding occurs because of integration or "pooling" of low-level features at a single, relatively early stage of visual processing. Recent studies suggest that crowding can occur also between high-level object representations. The most relevant findings come from studies with faces and may be specific to faces. We examined whether crowding can occur at the object configural level in addition to part-level crowding, using nonface objects. Target (a disconnected square or diamond made of four elements) identification was measured at varying eccentricities. The flankers were similar either to the target parts or to the target configuration. The results showed crowding in both cases: Flankers interfered with target identification such that identification accuracy decreased with an increase in eccentricity, and no interference was observed at the fovea. Crowding by object parts, however, was weaker and had smaller spatial extent than crowding by object configurations; we related this finding to the relationship between crowding and perceptual organization. These results provide strong evidence that crowding occurs not only between object parts but also between configural representations of objects.

Attending to illusory differences in object size

Focused visual attention can be shifted between objects and locations (attentional orienting) or expanded and contracted in spatial extent (attentional focusing). Although orienting and focusing both modulate visual processing, they have been shown to be distinct, independent modes of attentional control. Objects play a central role in visual attention, and it is known that high-level object representations guide attentional orienting. It not known, however, whether attentional focusing is driven by low-level object representations (which code object size in terms of retinotopic extent) or by high-level representations (which code perceived size). We manipulated the perceived size of physically identical objects by using line drawings or photographs that induced the Ponzo illusion, in a task requiring the detection of a target within these objects. The distribution of attention was determined by the perceived size and not by the retinotopic size of an attended object, indicating that attentional focusing is guided by high-level object representations.

A Real-World Size Organization of Object Responses in Occipitotemporal Cortex

While there are selective regions of occipitotemporal cortex that respond to faces, letters, and bodies, the large-scale neural organization of most object categories remains unknown. Here, we find that object representations can be differentiated along the ventral temporal cortex by their real-world size. In a functional neuroimaging experiment, observers were shown pictures of big and small real-world objects (e.g., table, bathtub; paperclip, cup), presented at the same retinal size. We observed a consistent medial-to-lateral organization of big and small object preferences in the ventral temporal cortex, mirrored along the lateral surface. Regions in the lateral-occipital, inferotemporal, and parahippocampal cortices showed strong peaks of differential real-world size selectivity and maintained these preferences over changes in retinal size and in mental imagery. These data demonstrate that the real-world size of objects can provide insight into the spatial topography of object representation.

Frequency-tagging object awareness

The rate of visual information processing in humans has been studied with different techniques. For example, the Steady-State Visual Evoked Potentials (SSVEP) method modulates stimulus luminance at a specific frequency, and reveals a corresponding “frequency tagging” at rates up to 10–20 Hz. Such luminance fluctuations, however, directly affect the earliest processing stages, whose influence cascades to the entire visual system. Hence it is impossible with classic SSVEP to distinguish neural correlates of high-level object representations (semantic content) from low-level activities. Here we present a novel technique called Semantic Wavelet-Induced Frequency Tagging (SWIFT) in which advanced image manipulation allows us to isolate object representations using frequency-tagging. By periodically scrambling the image in the wavelets domain we modulate its semantic content (object form), without disturbing local or global low-level attributes. Human observers (N = 16) watched sequences containing no real object, or objects that were either easily or difficultly detectable. Each trial consisted in two periods: a naive period where the subject saw the sequence for the first time (and sometimes did not consciously recognize the embedded object) and a cognizant period where the sequence was presented again after revealing the object identity. When no object was perceived (i.e., either the sequence contained no object or the object was not recognized), the evoked activity was no different from baseline; but whenever observers were aware of the semantic content a tagging response emerged. In a separate experiment (N = 24) we compared SWIFT with classic SSVEP at 8 tagging frequencies between 1.5 and 12 Hz. While classic SSVEP was insensitive to object-content, SWIFT responded only to sequences containing objects, and at frequencies up to ∼4 Hz. The SWIFT technique promises to be an elegant EEG method for tracking high-level activity –our first results indicate that the visual system can only form 3 to 4 distinct object representations per second.

Object-level visual information gets through the bottleneck of crowding

Natural visual scenes are cluttered. In such scenes, many objects in the periphery can be crowded, blocked from identification, simply because of the dense array of clutter. Outside of the fovea, crowding constitutes the fundamental limitation on object recognition and is thought to arise from the limited resolution of the neural mechanisms that select and bind visual features into coherent objects. Thus it is widely believed that in the visual processing stream, a crowded object is reduced to a collection of dismantled features with no surviving holistic properties. Here, we show that this is not so: an entire face can survive crowding and contribute its holistic attributes to the perceived average of the set, despite being blocked from recognition. Our results show that crowding does not dismantle high-level object representations to their component features.

Attention to Surfaces Modulates Motion Processing in Extrastriate Area MT

In the visual system, early atomized representations are grouped into higher-level entities through processes of perceptual organization. Here we present neurophysiological evidence that a representation of a simple object, a surface defined by color and motion, can be the unit of attentional selection at an early stage of visual processing. Monkeys were cued by the color of a fixation spot to attend to one of two transparent random-dot surfaces, one red and one green, which occupied the same region of space. Motion of the attended surface drove neurons in the middle temporal (MT) visual area more strongly than physically identical motion of the non-attended surface, even though both occurred within the spotlight of attention. Surface-based effects of attention persisted even without differential surface coloring, but attentional modulation was stronger with color. These results show that attention can select surface representations to modulate visual processing as early as cortical area MT.

Motion-based orienting, segmenting and tracking in a model of object learning

Most current computational models of object learning rely on labeled, pre-normalized collections of static images to guide the process of extracting features that are diagnostic of an object class. From the standpoint of the developing visual system, these approaches are deeply problematic. An infant has no reliable labels to attach to newly observed objects, there is no guarantee that objects will be observed at a consistent scale or orientation, and, an infant's visual world is dynamic rather than static. A growing body of results from Project Prakash, and also from studies with infants, points to the profound significance of motion information in the early development of visual object concepts. But, precisely how do dynamic cues contribute to object learning? Here we address this question in the context of a comprehensive model of object learning named ‘Dylan’ (Dynamic input based Learning in Artificial and Natural systems). Specifically, we show how dynamic information can be used to orient towards, segment and track potential object candidates, without the need for pre-normalization or labeling of input data. Interestingly, the three processes can be implemented via qualitatively similar computations involving clustering of motion vectors, operating over progressively longer time-scales. By tracking a candidate proto-object over many frames of a video sequence and recording its appearance continuously, we can construct a temporally extended model of that object's appearance. This series of images can then be used as the starting point for various feature extraction techniques, and to build up tolerances to variations of scale and illumination. Our system can also be selectively compromised to more closely approximate the perceptual limitations of the developing visual system, providing a useful means for understanding the influence of poor acuity and color sensitivity on high-level object representations.

The Integration of Scene Understanding within a Geographic Information System: A Prototype Approach for Agricultural Applications

This paper describes progress towards engineering a combined GIS and image understanding system, designed to analyse remotely‐sensed imagery in a task‐oriented manner, with the task and contextual parameters being supplied by the GIS. The overall aim is to evaluate some of the more recent developments in the field of computer vision and to show how they may be integrated (along with more traditional methods of image interpretation) to improve system performance in terms of object recognition and extraction. Three aspects are reported: (1) Development of a formal notation as the basis of describing the process of transforming data from low level image representations to high level object representations; (2) Design of collaborating low and high level image processes used in this abstraction process and their control via Process Networks encapsulated into frames; and (3) Use of Dempster‐Shafer Decision Theory as the basis for combining different types of evidence to improve the recognition of objects in the scene. Some sample results are presented to illustrate the issues raised and mechanisms for the co‐operative processing of data are suggested.