Natural Image Patches Research Articles

Exploiting Statistical Dependencies in Sparse Representations for Signal Recovery

Signal modeling lies at the core of numerous signal and image processing applications. A recent approach that has drawn considerable attention is sparse representation modeling, in which the signal is assumed to be generated as a combination of a few atoms from a given dictionary. In this work we consider a Bayesian setting and go beyond the classic assumption of independence between the atoms. The main goal of this paper is to introduce a statistical model that takes such dependencies into account and show how this model can be used for sparse signal recovery. We follow the suggestion of two recent works and assume that the sparsity pattern is modeled by a Boltzmann machine, a commonly used graphical model. For general dependency models, exact MAP and MMSE estimation of the sparse representation becomes computationally complex. To simplify the computations, we propose greedy approximations of the MAP and MMSE estimators. We then consider a special case in which exact MAP is feasible, by assuming that the dictionary is unitary and the dependency model corresponds to a certain sparse graph. Exploiting this structure, we develop an efficient message passing algorithm that recovers the underlying signal. When the model parameters defining the underlying graph are unknown, we suggest an algorithm that learns these parameters directly from the data, leading to an iterative scheme for adaptive sparse signal recovery. The effectiveness of our approach is demonstrated on real-life signals - patches of natural images - where we compare the denoising performance to that of previous recovery methods that do not exploit the statistical dependencies.

Building and using a database of one trillion natural-image patches.

Many example-based image processing algorithms operate on image patches (texture synthesis, resolution enhancement, image denoising, and so on). However, inaccessibility to a large, varied collection of image patches has hindered widespread adoption of these methods. The authors describe the construction of a database of one trillion image patches and demonstrate its research utility.

A Two-Layer Model of Natural Stimuli Estimated with Score Matching

We consider a hierarchical two-layer model of natural signals in which both layers are learned from the data. Estimation is accomplished by score matching, a recently proposed estimation principle for energy-based models. If the first-layer outputs are squared and the second-layer weights are constrained to be nonnegative, the model learns responses similar to complex cells in primary visual cortex from natural images. The second layer pools a small number of features with similar orientation and frequency, but differing in spatial phase. For speech data, we obtain analogous results. The model unifies previous extensions to independent component analysis such as subspace and topographic models and provides new evidence that localized, oriented, phase-invariant features reflect the statistical properties of natural image patches.

Optimal detection and estimation of defocus in natural images

Defocus signals are important in many aspects of vision including accommodation, the estimation of scale, distance, and depth, and the control of eye growth. However, little is known about the computations visual systems use to detect and estimate the magnitude of defocus under natural conditions. We investigated how to optimally estimate defocus blur in images of natural scenes, given the optical systems of primates. First, we selected a large set of well-focused natural image patches. Next, we filtered each image patch with point-spread functions derived from a wave-optics model of the primate (human) eye at different levels of defocus. Finally, we used a statistical learning method, based on Bayesian ideal observer theory, to determine the spatial-frequency filters that are optimal for estimating retinal image defocus in natural scenes. We found that near the center of the visual field, the optimal spatial-frequency filters form a systematic set that is concentrated in the range of 5-15 cyc/deg, the range that drives human accommodation. Furthermore, we found that the optimal filters can be closely approximated by a linear combination of a small number of difference-of-Gaussian filters. Cells with such center-surround receptive field structure are commonplace in the early visual system. Thus, retinal neurons sensitive to this frequency range should contribute strongly to the retinal and/or post-retinal mechanisms that detect and estimate defocus. The optimal filters were also used to detect, discriminate, and identify defocus levels for 1 deg natural image patches. Consistent with human psychophysical data, detection thresholds were higher than discrimination thresholds. Also, once defocus exceeds 0.25 diopters, we found that 0.25 diopter changes in defocus can be identified with better than 86% accuracy. The estimated optimal filters are biologically plausible and provide a rigorous starting point for developing principled hypotheses for the neural mechanisms that encode and exploit optical defocus signals.

Image Inpainting by Patch Propagation Using Patch Sparsity

This paper introduces a novel examplar-based inpainting algorithm through investigating the sparsity of natural image patches. Two novel concepts of sparsity at the patch level are proposed for modeling the patch priority and patch representation, which are two crucial steps for patch propagation in the examplar-based inpainting approach. First, patch structure sparsity is designed to measure the confidence of a patch located at the image structure (e.g., the edge or corner) by the sparseness of its nonzero similarities to the neighboring patches. The patch with larger structure sparsity will be assigned higher priority for further inpainting. Second, it is assumed that the patch to be filled can be represented by the sparse linear combination of candidate patches under the local patch consistency constraint in a framework of sparse representation. Compared with the traditional examplar-based inpainting approach, structure sparsity enables better discrimination of structure and texture, and the patch sparse representation forces the newly inpainted regions to be sharp and consistent with the surrounding textures. Experiments on synthetic and natural images show the advantages of the proposed approach.

All bits are welcome: a coherent model for synergistic generalization and discrimination

Event Abstract Back to Event All bits are welcome: a coherent model for synergistic generalization and discrimination András Lőrincz1* and Gábor Szirtes1 1 Eötvös Loránd University, Software Technology and Methodology, Hungary In modeling neural information processing, the ability to generalize and to discriminate are two desirable features, yet they seem to be in conflict. Based on recent results in signal processing we propose a two-stage representational model that learns to decompose the observations into a typical part serving generalization and a sparse form using an overcomplete dictionary. The latter form is useful in grasping peculiarities in a concise way, but even small perturbations can result in large variations. When these two representational forms are tuned in concert then both become more powerful: generalization gets more robust against outliers, while sparse coding can develop increasingly sparse overcomplete representations less affected by noise. On natural image patches our model simultaneously develops Difference-of-Gaussian like basis as preprocessing for the representation of the typicalities and Gabor filter like basis for representing sparse features of the inputs. In turn, the proposed model may provide a simple and coherent account on the hierarchical arrangement of the spatial receptive fields at the first stages of early visual processing (that is in the retina, LGN and primary visual cortex). The iterative algorithm can be implemented in a generative neural network form and it can be seen as a generalization and unification of other attractive proposals, like infomax, slow-feature analysis and sparse coding. We argue that the principles behind the proposed algorithm make it a central component in diverse hierarchical computations performed by neural systems. Keywords: Information Processing*, Learning, Memory, modelling, sensory processing Conference: Bernstein Conference on Computational Neuroscience, Berlin, Germany, 27 Sep - 1 Oct, 2010. Presentation Type: Presentation Topic: Bernstein Conference on Computational Neuroscience Citation: Lőrincz A and Szirtes G (2010). All bits are welcome: a coherent model for synergistic generalization and discrimination. Front. Comput. Neurosci. Conference Abstract: Bernstein Conference on Computational Neuroscience. doi: 10.3389/conf.fncom.2010.51.00012 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 01 Sep 2010; Published Online: 22 Sep 2010. * Correspondence: Dr. András Lőrincz, Eötvös Loránd University, Software Technology and Methodology, Budapest, Hungary, andras.lorincz@t-online.hu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers András Lőrincz Gábor Szirtes Google András Lőrincz Gábor Szirtes Google Scholar András Lőrincz Gábor Szirtes PubMed András Lőrincz Gábor Szirtes Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

A generative model of the covariance structure of images

Event Abstract Back to Event A generative model of the covariance structure of images Geoffrey E. Hinton1* 1 University of Toronto, Canada It is possible to model images using a Markov Random Field in which pairwise interactions between nearby pixels are used to define an energy function that favors smooth patches. A more powerful model, first suggested by Geman and Geman, is to use a gated MRF in which binary hidden variables determine the weights on the lateral connections between pixels. The states of these binary variables are easy to infer from an image and they provide a description of the image structure that is very useful for recognizing objects. I will describe a way to greatly reduce the number of parameters in such a model so that it can be learned fairly easily. The model can be made to learn nice topographic maps by using local connectivity between a layer of "simple" cells and a layer of "complex" cells. When applied to RGB patches of natural images, this new generative model learned a map that consisted mainly of fairly high-frequency gray-level filters that learned to be exactly balanced in the RGB channels. The map also contained a color blob consisting of low-frequency red-green and yellow-blue filters. The division of filters into spatially separate gray-level and color-opponent types does not seem to require any innate specification. It can be explained by the the 3-way interaction of a general unsupervised learning algorithm, local connectivity, and the statistical structure of natural images. This is joint work with Marc'Aurelio Ranzato. Conference: Computational and Systems Neuroscience 2010, Salt Lake City, UT, United States, 25 Feb - 2 Mar, 2010. Presentation Type: Oral Presentation Topic: Oral presentations Citation: Hinton GE (2010). A generative model of the covariance structure of images. Front. Neurosci. Conference Abstract: Computational and Systems Neuroscience 2010. doi: 10.3389/conf.fnins.2010.03.00037 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 18 Feb 2010; Published Online: 18 Feb 2010. * Correspondence: Geoffrey E Hinton, University of Toronto, Toronto, Canada, hinton@cs.toronto.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Geoffrey E Hinton Google Geoffrey E Hinton Google Scholar Geoffrey E Hinton PubMed Geoffrey E Hinton Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Receptive Field Self-Organization in a Model of the Fine Structure in V1 Cortical Columns

We study a dynamical model of processing and learning in the visual cortex, which reflects the anatomy of V1 cortical columns and properties of their neuronal receptive fields. Based on recent results on the fine-scale structure of columns in V1, we model the activity dynamics in subpopulations of excitatory neurons and their interaction with systems of inhibitory neurons. We find that a dynamical model based on these aspects of columnar anatomy can give rise to specific types of computations that result in self-organization of afferents to the column. For a given type of input, self-organization reliably extracts the basic input components represented by neuronal receptive fields. Self-organization is very noise tolerant and can robustly be applied to different types of input. To quantitatively analyze the system's component extraction capabilities, we use two standard benchmarks: the bars test and natural images. In the bars test, the system shows the highest noise robustness reported so far. If natural image patches are used as input, self-organization results in Gabor-like receptive fields. In quantitative comparison with in vivo measurements, we find that the obtained receptive fields capture statistical properties of V1 simple cells that algorithms such as independent component analysis or sparse coding do not reproduce.

Perception of differences in natural-image stimuli

Visual Difference Predictor (VDP) models have played a key role in digital image applications such as the development of image quality metrics. However, little attention has been paid to their applicability to peripheral vision. Central (i.e., foveal) vision is extremely sensitive for the contrast detection of simple stimuli such as sinusoidal gratings, but peripheral vision is less sensitive. Furthermore, crowding is a well-documented phenomenon whereby differences in suprathreshold peripherally viewed target objects (such as individual letters or patches of sinusoidal grating) become more difficult to discriminate when surrounded by other objects (flankers). We examine three factors that might influence the degree of crowding with natural-scene stimuli (cropped from photographs of natural scenes): (1) location in the visual field, (2) distance between target and flankers, and (3) flanker-target similarity. We ask how these factors affect crowding in a suprathreshold discrimination experiment where observers rate the perceived differences between two sequentially presented target patches of natural images. The targets might differ in the shape, size, arrangement, or color of items in the scenes. Changes in uncrowded peripheral targets are perceived to be less than for the same changes viewed foveally. Consistent with previous research on simple stimuli, we find that crowding in the periphery (but not in the fovea) reduces the magnitudes of perceived changes even further, especially when the flankers are closer and more similar to the target. We have tested VDP models based on the response behavior of neurons in visual cortex and the inhibitory interactions between them. The models do not explain the lower ratings for peripherally viewed changes even when the lower peripheral contrast sensitivity was accounted for; nor could they explain the effects of crowding, which others have suggested might arise from errors in the spatial localization of features in the peripheral image. This suggests that conventional VDP models do not port well to peripheral vision.

A Patch-Based Structural Masking Model with an Application to Compression

The ability of an image region to hide or mask a given target signal continues to play a key role in the design of numerous image processing and vision systems. However, current state-of-the-art models of visual masking have been optimized for artificial targets placed upon unnatural backgrounds. In this paper, we (1) measure the ability of natural-image patches in masking distortion; (2) analyze the performance of a widely accepted standard masking model in predicting these data; and (3) report optimal model parameters for different patch types (textures, structures, and edges). Our results reveal that the standard model of masking does not generalize across image type; rather, a proper model should be coupled with a classification scheme which can adapt the model parameters based on the type of content contained in local image patches. The utility of this adaptive approach is demonstrated via a spatially adaptive compression algorithm which employs patch-based classification. Despite the addition of extra side information and the high degree of spatial adaptivity, this approach yields an efficient wavelet compression strategy that can be combined with very accurate rate-control procedures.

Factorization and structure of natural <inline-formula><math altimg="none" display="inline" overflow="scroll"><mrow><mn>4</mn><mo>×</mo><mn>4</mn></mrow></math></inline-formula> patch densities

The objective of this work is to give an approximate statistical model of the distribution of 4×4 pixel natural image patches B. Given the huge size of the sample space, we collect the required statistics not directly over B, but, instead, over a fractal compression inspired representation of B, namely by a triplet (DB,µB,B), with B being the patch's contrast, µB its brightness, and DB a codebook representation of the mean-variance normalization of B:(B?µB)/B. While not coinciding exactly with the true natural patch density p(B), the density (B)=p(DB,µB,B) should give an adequate approximation of p(B), because BBDB+µB. Our first main result is a factorization of the probability density p(D,µ,) as p(D,µ,)p(D)p(µ)p()(||B||), with being a high-contrast correction. Here, the brightness term p(µ) is largely irrelevant, and our second main result deals with the structure of the other two factors, showing that p() follows an exponential distribution, and that p(D) is uniformly distributed with respect to volume in image space. These results are largely independent of the codebook used.

Discrimination of amplitude spectrum slope in the fovea and parafovea and the local amplitude distributions of natural scene imagery

A number of studies have investigated whether human visual performance can be related to the general form of the amplitude spectra (i.e., 1/f(alpha)) of natural scenes. Here, it is argued that there are some discrepancies in the data between some of those studies and that one possible explanation for the discrepancies may be related to differences in methodology (e.g., stimuli presented to the fovea as opposed to the parafovea). We sought to resolve some of the discrepancies with two psychophysical paradigms involving alpha discrimination with visual noise and natural scene image patches presented to the fovea or parafovea. Fovea-parafovea threshold differences were apparent for stimuli possessing alpha values < 1.0, with the parafovea typically showing highest thresholds for reference alpha values in the 0.74-0.85 range. Both fovea and parafovea thresholds were lowest in the 1.2-1.4 range. In addition, we conducted a local amplitude distribution analysis (i.e., assessed local alpha) with a large set of high-resolution natural scene imagery and found that the results of that analysis provided a better account of the alpha discrimination thresholds for stimuli presented to the fovea as opposed to the parafovea.

Local image structures and optic flow estimation

Different kinds of local image structures (such as homogeneous, edge-like and junction-like patches) can be distinguished by the intrinsic dimensionality of the local signals. Intrinsic dimensionality makes use of variance from a point and a line in spectral representation of the signal in order to classify it as homogeneous, edge-like or junction-like. The concept of intrinsic dimensionality has been mostly exercised using discrete formulations; however, recent work (; ) has introduced a continuous definition. The current study analyzes the distribution of local patches in natural images according to this continuous understanding of intrinsic dimensionality. This distribution reveals specific patterns than can be also associated to local image structures established in computer vision and which can be related to orientation and optic flow features. In particular, we link quantitative and qualitative properties of optic-flow error estimates to these patterns. In this way, we also introduce a new tool for better analysis of optic flow algorithms.

Centre‐surround interactions in response to natural scene stimulation in the primary visual cortex

Centre-surround interaction in the primary visual cortex (area V1) has been studied extensively using artificial, abstract stimulus patterns, such as bars, gratings and simple texture patterns. In this experiment, we extend the study of centre-surround interaction by using natural scene images. We systematically varied the contrast of natural image surrounds presented outside the classical receptive field (CRF), and recorded neuronal response to a natural image patch presented within the CRF in area V1 of awake, fixating macaques. For the majority of neurons (67 out of 111), the natural image surrounds profoundly modulated, mainly by suppressing, neuronal responses to CRF images. These modulatory effects started at the earliest stage of neuronal responses, and often depended on the contrast and higher-order structures of the surrounds. For 47 out of 67 neurons, randomising the phases of the Fourier spectrum of the natural image surround diminished the centre-surround interaction. Our results suggest that the centre-surround interaction in area V1 can be extended to natural vision, and is sensitive to the higher-order structures of natural scene images, such as image contours.

The Nonlinear Statistics of High-Contrast Patches in Natural Images

Recently, there has been a great deal of interest in modeling the non-Gaussian structures of natural images. However, despite the many advances in the direction of sparse coding and multi-resolution analysis, the full probability distribution of pixel values in a neighborhood has not yet been described. In this study, we explore the space of data points representing the values of 3 × 3 high-contrast patches from optical and 3D range images. We find that the distribution of data is extremely “sparse” with the majority of the data points concentrated in clusters and non-linear low-dimensional manifolds. Furthermore, a detailed study of probability densities allows us to systematically distinguish between images of different modalities (optical versus range), which otherwise display similar marginal distributions. Our work indicates the importance of studying the full probability distribution of natural images, not just marginals, and the need to understand the intrinsic dimensionality and nature of the data. We believe that object-like structures in the world and the sensor properties of the probing device generate observations that are concentrated along predictable shapes in state space. Our study of natural image statistics accounts for local geometries (such as edges) in natural scenes, but does not impose such strong assumptions on the data as independent components or sparse coding by linear change of bases.

Neural activity in areas V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing.

Under natural viewing conditions primates make frequent exploratory eye movements across complex scenes. We recorded neural activity of 62 cells in visual areas V1, V2 and V4 in an awake behaving monkey that freely viewed natural images. About half of the cells studied showed a modulation in firing rate following some of the eye movements made during free viewing, though the proportions showing a discernible modulation varied across areas. These cells were also examined under controlled viewing conditions in which gratings or natural image patches were flashed in and around the classical receptive field while the animal performed a fixation task. Activity rates were generally highest with flashed gratings and lowest during free viewing. Flashed natural image patches evoked responses between these two extremes, and the responses were higher when the patches were confined to the classical receptive field than when they extended into the non-classical surround. Thus the reduction of activity during free viewing relative to that obtained with flashed gratings is partly attributable to natural images being less effective stimuli and partly to suppressive spatio-temporal neural mechanisms that are important during natural vision.

Structure detection: a statistically certified unsupervised learning procedure.

We present a class of structure detection procedures (SDPs) that can extract the characteristic structures in an arbitrary population of images. An SDP adaptively augments the power of a novel, statistical, structure test to reject the null hypothesis that a randomly chosen image is devoid of structure. The core of the structure test consists of an orthonormal basis B of receptive fields that is refined into an increasingly sensitive detector of characteristic image structures. Adaptive refinement is accomplished as follows: for each image x in a random training sequence, B is updated by a planar rotation that decreases the p-value of a statistical structure test for x. This image-by-image refinement procedure is very efficient, obeying time and space constraints similar to those that limit processes of perceptual organization in real organisms. SDPs' capabilities are demonstrated in three test populations: natural images, faulty random number generators, and artificial images composed of mixtures of basis functions. (1) An SDP succeeds in rejecting the null hypothesis that the UNIX random number generator rand() is truly random. (2) When images are composed by adding arbitrary pairs of orthogonal component images, an SDP extracts the components. (3) For a large set of natural image patches, an SDP yields a basis B1 that detects structure with p-value < 0.005 in 88% of a new set of patches. B1's elements resemble the receptive fields of V1 simple cells. (4) Of special interest are biconvergent SDPs that derive in parallel a basis B, as well as a pointwise transformation f, specifically sensitized to evaluate the response values that result from applying B to images in the target population. A biconvergent SDP applied to natural image patches yields a basis B2 similar to B1, as well as a pointwise transformation f with vastly heightened sensitivity to extreme response values. We conjecture that sensory neurons have evolved cooperatively to maximize their collective power to reject the null hypothesis that their input is devoid of structure, thereby evolving receptive fields that efficiently represent characteristic input structures.