Models Of Visual Object Recognition Research Articles

Capturing the objects of vision with neural networks.

Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioural studies have documented how object representations emerge through grouping, amodal completion, proto-objects and object files. By contrast, deep neural network models of visual object recognition remain largely tethered to sensory input, despite achieving human-level performance at labelling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving the development of deep neural network models that will put the object into object recognition.

Orthogonal Representations of Object Shape and Category in Deep Convolutional Neural Networks and Human Visual Cortex

Deep Convolutional Neural Networks (CNNs) are gaining traction as the benchmark model of visual object recognition, with performance now surpassing humans. While CNNs can accurately assign one image to potentially thousands of categories, network performance could be the result of layers that are tuned to represent the visual shape of objects, rather than object category, since both are often confounded in natural images. Using two stimulus sets that explicitly dissociate shape from category, we correlate these two types of information with each layer of multiple CNNs. We also compare CNN output with fMRI activation along the human visual ventral stream by correlating artificial with neural representations. We find that CNNs encode category information independently from shape, peaking at the final fully connected layer in all tested CNN architectures. Comparing CNNs with fMRI brain data, early visual cortex (V1) and early layers of CNNs encode shape information. Anterior ventral temporal cortex encodes category information, which correlates best with the final layer of CNNs. The interaction between shape and category that is found along the human visual ventral pathway is echoed in multiple deep networks. Our results suggest CNNs represent category information independently from shape, much like the human visual system.

Adaptation in models of visual object recognition

Convolutional neural network (CNN) models of the ventral stream provide an unprecedented opportunity to relate neural mechanisms to sensory representations and even perception. Current CNNs lack the temporal dynamics of biological vision such as adaptation to previous stimulation. Perceptually, adaptation has been widely studied in the form of aftereffects (Webster, 2015), while in single neurons adaptation is often equated to repetition suppression (Vogels, 2016). Whereas the two are often thought to be associated, they remain to be integrated in a truly general framework. One proposed mechanism underlying repetition suppression is a reduced excitability depending on previous neural activity, called response fatigue. Here, we implemented fatigue in each unit of a CNN (Krizhevsky et al., 2012) and asked whether it could account for more complex phenomena of neural and visual adaptation. Specifically, we assigned a latent fatigue variable to each unit that increased after high, but decreased after lower activation. The activation of a unit was then obtained by subtracting its fatigue from its input activity (before the linear rectifier). The resulting CNN units showed repetition suppression matching neural adaptation on several hallmark properties: stimulus-specificity, increased adaptation in higher layers, adaptation degree proportional with the number of repetitions (Vinken et al., 2017), and decreased adaptation with longer interstimulus intervals (Sawamura et al., 2006). Furthermore, the response patterns could account for the perceptual effects we tested: from afterimages in the first layer to a face gender aftereffect in later layers (Webster et al., 2004). Thus, when considered in a CNN, a simple mechanism of response fatigue operating at the level of single neurons can account for complex adaptation effects. In addition to providing a general model for adaptation, these results demonstrate the strength of using deep neural networks to connect low-level canonical neural properties or computations to high-level neural and perceptual phenomena.

Non-accidental properties, metric invariance, and encoding by neurons in a model of ventral stream visual object recognition, VisNet.

When objects transform into different views, some properties are maintained, such as whether the edges are convex or concave, and these non-accidental properties are likely to be important in view-invariant object recognition. The metric properties, such as the degree of curvature, may change with different views, and are less likely to be useful in object recognition. It is shown that in a model of invariant visual object recognition in the ventral visual stream, VisNet, non-accidental properties are encoded much more than metric properties by neurons. Moreover, it is shown how with the temporal trace rule training in VisNet, non-accidental properties of objects become encoded by neurons, and how metric properties are treated invariantly. We also show how VisNet can generalize between different objects if they have the same non-accidental property, because the metric properties are likely to overlap. VisNet is a 4-layer unsupervised model of visual object recognition trained by competitive learning that utilizes a temporal trace learning rule to implement the learning of invariance using views that occur close together in time. A second crucial property of this model of object recognition is, when neurons in the level corresponding to the inferior temporal visual cortex respond selectively to objects, whether neurons in the intermediate layers can respond to combinations of features that may be parts of two or more objects. In an investigation using the four sides of a square presented in every possible combination, it was shown that even though different layer 4 neurons are tuned to encode each feature or feature combination orthogonally, neurons in the intermediate layers can respond to features or feature combinations present is several objects. This property is an important part of the way in which high capacity can be achieved in the four-layer ventral visual cortical pathway. These findings concerning non-accidental properties and the use of neurons in intermediate layers of the hierarchy help to emphasise fundamental underlying principles of the computations that may be implemented in the ventral cortical visual stream used in object recognition.

Complex Pattern Selectivity in Macaque Primary Visual Cortex Revealed by Large-Scale Two-Photon Imaging

Visual objects contain rich local high-order patternssuch as curvature, corners, and junctions. In the standard hierarchical model of visual object recognition, V1 neurons were commonly assumed to code local orientation components of those high-order patterns. Here, by using two-photon imaging in awake macaques and systematically characterizing V1 neuronal responses to an extensive set of stimuli, we found a large percentage of neurons in the V1 superficial layer responded more strongly to complex patterns, such as corners, junctions, and curvature, than to their oriented line or edge components. Our results suggest that those individual V1 neurons could play the role in detecting local high-order visual patterns in the early stage of object recognition hierarchy.

Are face representations depth cue invariant?

The visual system can process three-dimensional depth cues defining surfaces of objects, but it is unclear whether such information contributes to complex object recognition, including face recognition. The processing of different depth cues involves both dorsal and ventral visual pathways. We investigated whether facial surfaces defined by individual depth cues resulted in meaningful face representations-representations that maintain the relationship between the population of faces as defined in a multidimensional face space. We measured face identity aftereffects for facial surfaces defined by individual depth cues (Experiments 1 and 2) and tested whether the aftereffect transfers across depth cues (Experiments 3 and 4). Facial surfaces and their morphs to the average face were defined purely by one of shading, texture, motion, or binocular disparity. We obtained identification thresholds for matched (matched identity between adapting and test stimuli), non-matched (non-matched identity between adapting and test stimuli), and no-adaptation (showing only the test stimuli) conditions for each cue and across different depth cues. We found robust face identity aftereffect in both experiments. Our results suggest that depth cues do contribute to forming meaningful face representations that are depth cue invariant. Depth cue invariance would require integration of information across different areas and different pathways for object recognition, and this in turn has important implications for cortical models of visual object recognition.

Atoms of recognition in human and computer vision

Discovering the visual features and representations used by the brain to recognize objects is a central problem in the study of vision. Recently, neural network models of visual object recognition, including biological and deep network models, have shown remarkable progress and have begun to rival human performance in some challenging tasks. These models are trained on image examples and learn to extract features and representations and to use them for categorization. It remains unclear, however, whether the representations and learning processes discovered by current models are similar to those used by the human visual system. Here we show, by introducing and using minimal recognizable images, that the human visual system uses features and processes that are not used by current models and that are critical for recognition. We found by psychophysical studies that at the level of minimal recognizable images a minute change in the image can have a drastic effect on recognition, thus identifying features that are critical for the task. Simulations then showed that current models cannot explain this sensitivity to precise feature configurations and, more generally, do not learn to recognize minimal images at a human level. The role of the features shown here is revealed uniquely at the minimal level, where the contribution of each feature is essential. A full understanding of the learning and use of such features will extend our understanding of visual recognition and its cortical mechanisms and will enhance the capacity of computational models to learn from visual experience and to deal with recognition and detailed image interpretation.

V4 Neural Network Model for Shape-Based Feature Extraction and Object Discrimination

Visual area V4 plays an important role in the neural mechanism of shape recognition. V4 neurons exhibit selectivity for the orientation and curvature of boundary fragments. In this paper, we propose a novel neural network model of V4 for shape-based feature extraction and other vision tasks. The low-level layers of the model consist of computational units simulating simple cells and complex cells in the primary visual cortex. These layers extract preliminary visual features including edges and orientations. The V4 computational units calculate the entropy of the extracted features as a measure of visual saliency. The features around salient points are then selected and encoded with a layer of restricted Boltzmann machine to generate an intermediate representation of object shapes. The model is evaluated in shape distinction, feature detection, feature matching, and object discrimination experiments. The results demonstrate that this model generates discriminative local representation of object shapes. It shows a successful attempt to construct a computation model of visual object recognition in the brain.

Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance.

To go beyond qualitative models of the biological substrate of object recognition, we ask: can a single ventral stream neuronal linking hypothesis quantitatively account for core object recognition performance over a broad range of tasks? We measured human performance in 64 object recognition tests using thousands of challenging images that explore shape similarity and identity preserving object variation. We then used multielectrode arrays to measure neuronal population responses to those same images in visual areas V4 and inferior temporal (IT) cortex of monkeys and simulated V1 population responses. We tested leading candidate linking hypotheses and control hypotheses, each postulating how ventral stream neuronal responses underlie object recognition behavior. Specifically, for each hypothesis, we computed the predicted performance on the 64 tests and compared it with the measured pattern of human performance. All tested hypotheses based on low- and mid-level visually evoked activity (pixels, V1, and V4) were very poor predictors of the human behavioral pattern. However, simple learned weighted sums of distributed average IT firing rates exactly predicted the behavioral pattern. More elaborate linking hypotheses relying on IT trial-by-trial correlational structure, finer IT temporal codes, or ones that strictly respect the known spatial substructures of IT ("face patches") did not improve predictive power. Although these results do not reject those more elaborate hypotheses, they suggest a simple, sufficient quantitative model: each object recognition task is learned from the spatially distributed mean firing rates (100 ms) of ∼60,000 IT neurons and is executed as a simple weighted sum of those firing rates. Significance statement: We sought to go beyond qualitative models of visual object recognition and determine whether a single neuronal linking hypothesis can quantitatively account for core object recognition behavior. To achieve this, we designed a database of images for evaluating object recognition performance. We used multielectrode arrays to characterize hundreds of neurons in the visual ventral stream of nonhuman primates and measured the object recognition performance of >100 human observers. Remarkably, we found that simple learned weighted sums of firing rates of neurons in monkey inferior temporal (IT) cortex accurately predicted human performance. Although previous work led us to expect that IT would outperform V4, we were surprised by the quantitative precision with which simple IT-based linking hypotheses accounted for human behavior.

Fast neuromimetic object recognition using FPGA outperforms GPU implementations.

Recognition of objects in still images has traditionally been regarded as a difficult computational problem. Although modern automated methods for visual object recognition have achieved steadily increasing recognition accuracy, even the most advanced computational vision approaches are unable to obtain performance equal to that of humans. This has led to the creation of many biologically inspired models of visual object recognition, among them the hierarchical model and X (HMAX) model. HMAX is traditionally known to achieve high accuracy in visual object recognition tasks at the expense of significant computational complexity. Increasing complexity, in turn, increases computation time, reducing the number of images that can be processed per unit time. In this paper we describe how the computationally intensive and biologically inspired HMAX model for visual object recognition can be modified for implementation on a commercial field-programmable aate Array, specifically the Xilinx Virtex 6 ML605 evaluation board with XC6VLX240T FPGA. We show that with minor modifications to the traditional HMAX model we can perform recognition on images of size 128 × 128 pixels at a rate of 190 images per second with a less than 1% loss in recognition accuracy in both binary and multiclass visual object recognition tasks.

Non-Classical Connectionist Models of Visual Object Recognition

Dear Editor,It is with great interest that we read the article by Tayet al. on face recognition with quantum associative net-works, which was recently published in Cognitive Compu-tation [1]. In a series of computer simulations, the authorsdemonstrated how non-Turing-based quantum computingcan be harnessed for viewpoint-invariant face recognitionusing Hebb-like storage of image-encoding Gabor waveletsimplemented in a quantum-holographic procedure. Thepresented model was largely motivated by Hopfield’s neuralnetwork [2], which was then transformed into a quantum-holographic process in which the Hebbian memory wasreplaced by multiple self-interferences of quantum planewaves [3].This transformation was achieved by means of a simplesubstitution (see Table 1 for details) of Hopfield’s real-valued variables by the complex-valued variables changingas sinusoids (waves), while simultaneously preserving allinput-to-output transformations in the model. By doingthis, it becomes possible to combine the area of artificialneural networks with that of quantum computing in a rathernatural way. In particular, the authors showed that thequality of reconstructed images can be enhanced by meansof an iterative sampling method using (1) one optimumsampling frequency for which the best distinction amongimages is warranted and (2) by selecting Gabor coefficientswith least activity for correlation in the associative net-work, which is necessary to determine the order parametersin the model.Although the authors meticulously dealt with issuesrelated to quality improvements of reconstructed images,they made much less effort to address two essential issuesintimately linked to their work. First, while a substantialportion of the paper was dedicated to Gabor wavelets asbiologically plausible descriptors of receptive fields, itremains unclear how the implemented quantum-holo-graphic procedure—the central aspect of the introducedmodel—relates to current neurocognitive theories of visualinformation processing. Second, it is hard to see how thepresented results could differ from those yielded by pre-vious versions of quantum neural network models (see e.g.,[4, 5]) and especially, from the performance of classi-cal, that is, non-quantum neural networks for imagerecognition.Non-classical, i.e., quantum-like approaches to cognitionand its underlying neural substrates have attracted muchrecent attention [6–11]. While some areas such as mathe-matical psychology have benefited from alternative non-classical models of cognition [10, 12, 13], the relevance ofquantum neural models [14] and the actual nature of therelationship between quantum and neurophysiologicalprocesses [15, 16] in the wet and 300 Kelvin warm braintissue have remained largely elusive [17], as argumentsfrom both biophysics and computational neurosciencespeak against a possibility of quantum-like neural-levelprocesses [18, 19]. It is therefore far from clear how thepresented [1], neurobiologically implausible quantum-holographic procedure for image storage and recall shouldactually interact with a biologically plausible, non-quantum(Gabor-wavelet) component and at which level should suchinteractions occur in the human brain?

Towards a Cortical Prosthesis: Implementing A Spike-Based HMAX Model of Visual Object Recognition in Silico

Object recognition and categorization are computationally difficult tasks that are performed effortlessly by humans. Attempts have been made to emulate the computations in different parts of the primate cortex to gain a better understanding of the cortex and to design brain–machine interfaces that speak the same language as the brain. The HMAX model proposed by Riesenhuber and Poggio and extended by Serre <etal xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> attempts to truly model the visual cortex. In this paper, we provide a spike-based implementation of the HMAX model, demonstrating its ability to perform biologically-plausible MAX computations as well as classify basic shapes. The spike-based model consists of 2514 neurons and 17 <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\thinspace$</tex> </formula> 305 synapses (S1 Layer: 576 neurons and 7488 synapses, C1 Layer: 720 neurons and 2880 synapses, S2 Layer: 576 neurons and 1152 synapses, C2 Layer: 640 neurons and 5760 synapses, and Classifier: 2 neurons and 25 synapses). Without the limits of the retina model, it will take the system 2 min to recognize rectangles and triangles in 24 <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\,\times\,$</tex> </formula> 24 pixel images. This can be reduced to 4.8 s by rearranging the lookup table so that neurons which have similar responses to the same input(s) can be placed on the same row and affected in parallel.

Spared structural knowledge in a case of semantic dementia: Implications for models of object recognition, semantic memory and structural description

Most models of visual object recognition assume that items belonging to known categories are represented in long-term memory in terms of both structural descriptions and semantic representations. The former specify the visual appearance of category members and the latter allow recognising them as meaningful objects. Nevertheless, the format of these two kinds of representations and their relationships are still a matter of debate. Recently, the independence of structural and semantic representations has been questioned on the basis of the finding of an impaired performance of subjects suffering from semantic dementia on the object decision task, which was originally devised to tap the structural description system. In the present case study of a patient with semantic dementia, we provide data supporting the independence of these two systems. Our results allowed us to better qualify the content and format of structural descriptions in terms of purely geometric non-verbalizable information, specifying the appearance of exemplars at a rather coarse level of categorisation.

The Timing of Visual Object Categorization

An object can be categorized at different levels of abstraction: as natural or man-made, animal or plant, bird or dog, or as a Northern Cardinal or Pyrrhuloxia. There has been growing interest in understanding how quickly categorizations at different levels are made and how the timing of those perceptual decisions changes with experience. We specifically contrast two perspectives on the timing of object categorization at different levels of abstraction. By one account, the relative timing implies a relative timing of stages of visual processing that are tied to particular levels of object categorization: Fast categorizations are fast because they precede other categorizations within the visual processing hierarchy. By another account, the relative timing reflects when perceptual features are available over time and the quality of perceptual evidence used to drive a perceptual decision process: Fast simply means fast, it does not mean first. Understanding the short-term and long-term temporal dynamics of object categorizations is key to developing computational models of visual object recognition. We briefly review a number of models of object categorization and outline how they explain the timing of visual object categorization at different levels of abstraction.

Are cortical models really bound by the "binding problem"?

Are cortical models really bound by the "binding problem"?

Visual object agnosia without prosopagnosia or alexia: Evidence for hierarchical theories of visual recognition

Abstract A single case study is presented of a patient, Mr. W, with a selective deficit in recognizing pictures and real objects, linked to impaired stored visual knowledge about objects. Despite this, Mr. W maintained a preserved ability both to read aloud printed words and to recognize famous faces, when compared with age-matched control subjects. In addition, his access to semantic information from words was superior to that from pictures. The data provide evidence that visual agnosia can occur without alexia or prosopagnosia, contrary to recent proposals (Farah 1990, 1991). This finding is consistent with a hierarchical model of visual object recognition in which agnosia can reflect impaired stored knowledge of objects without accompanying perceptual deficits. The selective recognition deficit for objects further indicates that stored knowledge concerning different classes of visual stimuli (common objects, faces, and words) is separately represented in the brain.

Local and global contextual constraints on the identification of objects in scenes.

Objects likely to appear in a given real-world scene are frequently found to be easier to recognize. Two different sources of contextual information have been proposed as the basis for this effect: global scene background and individual companion objects. The present paper examines the relative importance of these two elements in explaining the context-sensitivity of object identification in full scenes. Specific sequences of object fixations were elicited during free scene exploration, while fixation times on designated target objects were recorded as a measure of ease of target identification. Episodic consistency between the target, the global scene background, and the object fixated just prior to the target (the prime), were manipulated orthogonally. Target fixation times were examined for effects of prime and background. Analyses show effects of both factors, which are modulated by the chronology and spatial extent of scene exploration. The results are discussed in terms of their implications for a model of visual object recognition in the context of real-world scenes.