Images Of Real-world Scenes Research Articles

Adding Shape to Saliency: A Proto-object Saliency Map for Predicting Fixations during Scene Viewing

Traditional saliency models predict fixations during scene viewing by computing local contrast between low-level color, intensity and orientation features; the higher the summed contrast the greater the probability of fixation. Evidence also suggests that high-level properties of objects are predictive of fixation locations. We attempt fixation prediction from proto-objects (POs), a mid-level representation existing between features and objects. Using our previously-reported proto-object model (Yu et. al., 2014, JoV), we segmented 384 images of real-world scenes into proto-objects, fragments of visual space, at multiple resolutions (feature-space bandwidths). We then built from these segmentations a saliency map by computing feature contrast between each proto-object and its local neighbors using intensity, color, orientation, and now, size and shape features. Center-surround size contrast was computed by comparing pixel area between a given proto-object and each "surrounding" neighbor. To compute shape contrast we first normalized a proto-object and a neighbor to have the same area, aligned them based on maximum area-overlap, then counted the number of pixels in the overlapping area (divided by the union of the areas), with a smaller overlap over neighbors coding a higher contrast. Doing this relative to each proto-object, then combining contrast signals across features and resolutions, generates a proto-object saliency map, which we used to predict the fixation behavior of 12 participants freely viewing the same scenes (each for 3 seconds) in anticipation of a memory test. We found that our proto-object saliency map predicted fixations as well or better than an Itti-Koch saliency model, and was nearly as predictive as the upper-limit defined by a Subject model obtained using the leave-one-out method. We conclude that size and shape features, quantified in terms of proto-objects, are used to guide overt visual attention, and that saliency-based models of fixation prediction need to recognize the importance of these mid-level visual features. Meeting abstract presented at VSS 2016

How Beauty Determines Gaze! Facial Attractiveness and Gaze Duration in Images of Real World Scenes.

We showed that the looking time spent on faces is a valid covariate of beauty by testing the relation between facial attractiveness and gaze behavior. We presented natural scenes which always pictured two people, encompassing a wide range of facial attractiveness. Employing measurements of eye movements in a free viewing paradigm, we found a linear relation between facial attractiveness and gaze behavior: The more attractive the face, the longer and the more often it was looked at. In line with evolutionary approaches, the positive relation was particularly pronounced when participants viewed other sex faces.

The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes.

We introduce the Clutterpalette, an interactive tool for detailing indoor scenes with small-scale items. When the user points to a location in the scene, the Clutterpalette suggests detail items for that location. In order to present appropriate suggestions, the Clutterpalette is trained on a dataset of images of real-world scenes, annotated with support relations. Our experiments demonstrate that the adaptive suggestions presented by the Clutterpalette increase modeling speed and enhance the realism of indoor scenes.

Method and experiments of subliminal cueing for real-world images

Unconscious attention shift triggered by a subliminal cue has been shown to be automatic; however, whether it can be brought into effect for images of real-world scenes remains to be investigated. We present a subliminal cueing method that flashes briefly a visual cue before presenting a real-world image to the viewer. The effectiveness of the method is verified by experiments using three types of cues (spatial cue, face cue, and object cue) of varied durations. Results show that depending on the cue type, the viewer's visual attention is directed to the cued visual hemifield or the cued location without engaging the viewer's awareness. The experiments demonstrate that a brief subliminal cue presented prior to the color image of a real-world complex scene can attract human visual attention. The method is useful for many applications that require efficient, unresisting attention shift to a target image area.

Eye gaze tracking reveals heightened attention to food in adults with binge eating when viewing images of real-world scenes

Individuals with eating disorders often exhibit food-related biases in attention tasks. To assess the engagement and maintenance of attention to food in adults with binge eating, in the present study, eye gaze tracking was used to compare fixations to food among non-clinical adults with versus without binge eating while they viewed images of real-world scenes. Fifty-seven participants' eye fixations were tracked and recorded throughout 8-second presentations of scenes containing high-calorie and/or low-caloriefood items in various settings (restaurants, social gatherings, etc.). Participants with binge eating fixated on both high-calorie and low-calorie food items significantly more than controls, and this was the case when the high- and low-calorie food items were presented in the same image and in different images. Participants with binge eating also fixated on food items significantly earlier in the presentations. A time course analysis that divided each 8-second presentation into 2-second intervals revealed that participants with binge eating attended to food items more than control participants throughout the 8-second presentation. These results have implications for theory regarding the initiation and maintenance of binge eating.

Blur detection is unaffected by cognitive load

Blur detection is affected by retinal eccentricity, but is it also affected by attentional resources? Research showing effects of selective attention on acuity and contrast sensitivity suggests that allocating attention should increase blur detection. However, research showing that blur affects selection of saccade targets suggests that blur detection may be pre-attentive. To investigate this question, we carried out experiments in which viewers detected blur in real-world scenes under varying levels of cognitive load manipulated by the N-back task. We used adaptive threshold estimation to measure blur detection thresholds at 0°, 3°, 6°, and 9° eccentricity. Participants carried out blur detection as a single task, a single task with to-be-ignored letters, or an N-back task with four levels of cognitive load (0, 1, 2, or 3-back). In Experiment 1, blur was presented gaze-contingently for occasional single eye fixations while participants viewed scenes in preparation for an easy picture recognition memory task, and the N-back stimuli were presented auditorily. The results for three participants showed a large effect of retinal eccentricity on blur thresholds, significant effects of N-back level on N-back performance, scene recognition memory, and gaze dispersion, but no effect of N-back level on blur thresholds. In Experiment 2, we replicated Experiment 1 but presented the images tachistoscopically for 200 ms (half with, half without blur), to determine whether gaze-contingent blur presentation in Experiment 1 had produced attentional capture by blur onset during a fixation, thus eliminating any effect of cognitive load on blur detection. The results with three new participants replicated those of Experiment 1, indicating that the use of gaze-contingent blur presentation could not explain the lack of effect of cognitive load on blur detection. Thus, apparently blur detection in real-world scene images is unaffected by attentional resources, as manipulated by the cognitive load produced by the N-back task.

Semantic descriptor ranking: a quantitative method for evaluating qualitative verbal reports of visual cognition in the laboratory or the clinic.

For scientific, clinical, and machine learning purposes alike, it is desirable to quantify the verbal reports of high-level visual percepts. Methods to do this simply do not exist at present. Here we propose a novel methodological principle to help fill this gap, and provide empirical evidence designed to serve as the initial “proof” of this principle. In the proposed method, subjects view images of real-world scenes and describe, in their own words, what they saw. The verbal description is independently evaluated by several evaluators. Each evaluator assigns a rank score to the subject’s description of each visual object in each image using a novel ranking principle, which takes advantage of the well-known fact that semantic descriptions of real life objects and scenes can usually be rank-ordered. Thus, for instance, “animal,” “dog,” and “retriever” can be regarded as increasingly finer-level, and therefore higher ranking, descriptions of a given object. These numeric scores can preserve the richness of the original verbal description, and can be subsequently evaluated using conventional statistical procedures. We describe an exemplar implementation of this method and empirical data that show its feasibility. With appropriate future standardization and validation, this novel method can serve as an important tool to help quantify the subjective experience of the visual world. In addition to being a novel, potentially powerful testing tool, our method also represents, to our knowledge, the only available method for numerically representing verbal accounts of real-world experience. Given that its minimal requirements, i.e., a verbal description and the ground truth that elicited the description, our method has a wide variety of potential real-world applications.

High dynamic range image deghosting by fast approximate background modelling

High Dynamic Range (HDR) images of real world scenes often suffer from ghosting artifacts caused by motion in the scene. Existing solutions to this problem typically either only address specific types of ghosting, or are very computationally expensive.We address ghosting by performing change detection on exposure-normalized images, then reducing the contribution of moving objects to the final composite on a frame-by-frame basis. Change detection is computationally advantageous and it can be applied to images exhibiting varied ghosting artifacts. We demonstrate our method both for Low Dynamic Range (LDR) and HDR images. Additional constraints based on a priori knowledge of the changing exposures apply to HDR images. We increase the stability of our approach by using recent superpixel segmentation techniques to enhance the change detection. Our solution includes a novel approach for areas that see motion throughout the capture, e.g., foliage blowing in the wind.We demonstrate the success of our approach on challenging ghosting scenarios, and that our results are comparable to existing state-of- the-art methods, while providing computational savings over these methods.

Robust and Precise Registration of Oblique Images Based on Scale-Invariant Feature Transformation Algorithm

The automatic registration of oblique images taken at different viewpoints remains a challenge until today. Based on scale-invariant feature transformation (SIFT) algorithm, a robust and accurate weighted least square matching (LSM) (SIFT/LSM) method modeled using 2-D projective transformation is proposed for highly accurate registration of oblique images. Normalized cross correlation (NCC) metric modified by an adaptive scale and orientation of SIFT features (SIFT/NCC) is proposed to obtain a good initial estimation for the SIFT/LSM. For practical use, image matching is implemented using a coarse-to-fine multistage strategy by sequentially incorporating the standard SIFT algorithm, SIFT/NCC, and SIFT/LSM. Experiments conducted on oblique images of real-world scenes demonstrate the feasibility of the proposed approach.

Deficits in Perception of Images of Real-World Scenes in Patients With a History of Amblyopia

Deficits in Perception of Images of Real-World Scenes in Patients With a History of Amblyopia

Functions of parahippocampal place area and retrosplenial cortex in real-world scene analysis: An fMRI study

We used functional MRI to investigate several hypotheses concerning the functions of posterior parahippocampal cortex and retrosplenial cortex, two regions that preferentially activate to images of real-world scenes compared to images of other meaningful visual stimuli such as objects and faces. We compared activation resulting from photographs of rooms, city streets, cityscapes, and landscapes against activation to a control condition of objects. Activation in posterior parahippocampal cortex, including parahippocampal place area, was greater for all scene types than objects, and greater for scenes that clearly convey information about local three-dimensional (3-D) structure (city streets and rooms) than scenes that do not (cityscapes and landscapes). Similar differences were observed in retrosplenial cortex, though activation was also greater for city streets than rooms. These results suggest that activation in both cortical areas is primarily related to analysis or representation of local 3-D space. The results are not consistent with hypotheses that these areas reflect panoramic spatial volume, an artificial versus natural category distinction, an indoor versus outdoor distinction, or the number of explicit objects depicted in a scene image.

Deficits in Perception of Images of Real-World Scenes in Patients With a History of Amblyopia

To investigate the perception of images of real-world scenes in patients with amblyopia and to compare their performance with that of visually normal participants by viewing conditions (monocular vs binocular) and by treatment outcomes (successfully vs unsuccessfully treated vs normal eyes). Thirty-nine healthy and 26 amblyopic individuals who had undergone previous amblyopia treatment were recruited to perform a match-to-sample task that used images of real-world scenes. Rates of correct, incorrect, and no responses and mean reaction time were recorded. Performance during monocular viewing showed that the mean correct response rate was 59% in the amblyopic eyes, 62% in the fellow eyes, and 67% in the normal eyes (P=.008). During binocular viewing, the correct response rate remained reduced at 58% in amblyopic patients compared with 68% in participants with normal vision (P=.03). Performance by treatment outcomes showed that the mean correct response rate was 59% in the unsuccessfully treated group, 64% in the successfully treated group, and 67% in the normal group (P=.002). There was no difference in performance among amblyopia subtypes. Real-world scene perception is impaired in amblyopia, with the poorest performance during amblyopic monocular and binocular viewing. Despite successful treatment of the amblyopic eye to normal acuity levels, perception of images in real-world scenes remains deficient in patients with a history of amblyopia.

Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum

We propose the concept of a generalized assorted pixel (GAP) camera, which enables the user to capture a single image of a scene and, after the fact, control the tradeoff between spatial resolution, dynamic range and spectral detail. The GAP camera uses a complex array (or mosaic) of color filters. A major problem with using such an array is that the captured image is severely under-sampled for at least some of the filter types. This leads to reconstructed images with strong aliasing. We make four contributions in this paper: 1) we present a comprehensive optimization method to arrive at the spatial and spectral layout of the color filter array of a GAP camera. 2) We develop a novel algorithm for reconstructing the under-sampled channels of the image while minimizing aliasing artifacts. 3) We demonstrate how the user can capture a single image and then control the tradeoff of spatial resolution to generate a variety of images, including monochrome, high dynamic range (HDR) monochrome, RGB, HDR RGB, and multispectral images. 4) Finally, the performance of our GAP camera has been verified using extensive simulations that use multispectral images of real world scenes. A large database of these multispectral images has been made available at http://www1.cs.columbia.edu/CAVE/projects/gap_camera/ for use by the research community.

Cortical activation to indoor versus outdoor scenes: an fMRI study.

Prior studies identify two cortical areas, posterior parahippocampal cortex and retrosplenial cortex, that preferentially activate to images of real-world scenes compared to images of other meaningful visual stimuli such as objects and faces. Behavioral and computational studies suggest that sub-categories of real-world scenes differ in their visual and semantic properties. It is presently unknown whether the cortical areas that have been implicated in scene analysis similarly activate differentially to behaviorally relevant scene sub-categories. To examine this issue, we directly compared cortical activation to indoor and outdoor scenes in an fMRI study with a large number of non-repeated images in each condition. Activation in posterior parahippocampal cortex, including parahippocampal place area, was significantly greater for indoor than outdoor scenes. In contrast, no such difference was observed in retrosplenial cortex, though this region preferentially activated to scenes over faces. These findings suggest differences in function in these two areas. The results are consistent with the view that posterior parahippocampal cortex is functional in processing local space.

The separation of reflected and transparent layers from real-world image sequence

This paper presents an optimal method for the separation of reflected and transparent layers from real-world scene images. Whereas past research has been applied to indoor environments and static cameras, our technique can be used for outdoor scenes and motion cameras. The method is based on spatio-temporal analysis, especially using epipolar plane images (EPI). The edge and color information of EPI has been used to segment the areas on EPIs efficiently and separate the reflected and transparent layers. This method can be used for refining building textures by removing reflections from captured images for the purpose of city modeling.

The perceived contrast of texture patches embedded in natural images

The visibility of an isolated simple stimulus is known to depend on its contrast. However, when such a stimulus is surrounded by other geometrically-simple stimuli, its perceived contrast can change markedly. Here, we examined whether such effects contribute to our perception of contrasts when we view real world scenes. We show that the perceived contrast of a luminance texture patch is suppressed when it is surrounded by images of real world scenes. We also show that the amount of this suppression depends on the spatial statistics of the surrounding images. We manipulated the second-order statistics of the images and found minimal suppression of perceived contrast at “un-natural” image statistics and maximal suppression at the characteristic statistics of natural images. This suggests that contrast gain control mechanisms in our visual system are optimally engaged when we view real world images.

Static scene analysis for the perception of heading.

In the current study, we explored observers' use of two distinct analyses for determining their direction of motion, or heading: a scene-based analysis and a motion-based analysis. In two experiments, subjects viewed sequentially presented, paired digitized images of real-world scenes and judged the direction of heading; the pairs were presented with various interstimulus intervals (ISIs). In Experiment 1, subjects could determine heading when the two frames were separated with a 1,000-ms ISI, long enough to eliminate apparent motion. In Experiment 2, subjects performed two tasks, a path-of-motion task and a memory-load task, under three different ISIs, 50 ms, 500 ms, and 1,000 ms. Heading accuracy decreased with an increase in ISI. Increasing memory load influenced heading judgments only for the longer ISI when motion-based information was not available. These results are consistent with the hypothesis that the scene-based analysis has a coarse spatial representation, is a sustained temporal process, and is capacity limited, whereas the motion-based analysis has a fine spatial resolution, is a transient temporal process, and is capacity unlimited.

The Dynamic Representation of Scenes

One of the more powerful impressions created by vision is that of a coherent, richly detailed world where everything is present simultaneously. Indeed, this impression is so compelling that we tend to ascribe these properties not only to the external world, but to our internal representations as well. But results from several recent experiments argue against this latter ascription. For example, changes in images of real-world scenes often go unnoticed when made during a saccade, flicker, blink, or movie cut. This “change blindness” provides strong evidence against the idea that our brains contain a picture-like representation of the scene that is everywhere detailed and coherent. How then do we represent a scene? It is argued here that focused attention provides spatiotemporal coherence for the stable representation of one object at a time. It is then argued that the allocation of attention can be co-ordinated to create a “virtual representation”. In such a scheme, a stable object representation is formed whenever needed, making it appear to higher levels as if all objects in the scene are represented in detail simultaneously.

Texture mapping 3D models of real-world scenes

Texture mapping has become a popular tool in the computer graphics industry in the last few years because it is an easy way to achieve a high degree of realism in computer-generated imagery with very little effort. Over the last decade, texture-mapping techniques have advanced to the point where it is possible to generate real-time perspective simulations of real-world areas by texture mapping every object surface with texture from photographic images of these real-world areas. The techniques for generating such perspective transformations are variations on traditional texture mapping that in some circles have become known as the Image Perspective Transformation or IPT technology. This article first presents a background survey of traditional texture mapping. It then continues with a description of the texture-mapping variations that achieve these perspective transformations of photographic images of real-world scenes. The style of the presentation is that of a resource survey rather thatn an in-depth analysis.

Information in the neuronal representation of individual stimuli in the primate temporal visual cortex.

To analyze the information provided about individual visual stimuli in the responses of single neurons in the primate temporal lobe visual cortex, neuronal responses to a set of 65 visual stimuli were recorded in macaques performing a visual fixation task and analyzed using information theoretical measures. The population of neurons analyzed responded primarily to faces. The stimuli included 23 faces and 42 nonface images of real-world scenes, so that the function of this brain region could be analyzed when it was processing relatively natural scenes. It was found that for the majority of the neurons significant amounts of information were reflected about which of several of the 23 faces had been seen. Thus the representation was not local, for in a local representation almost all the information available can be obtained when the single stimulus to which the neuron responds best is shown. It is shown that the information available about any one stimulus depended on how different (for example, how many standard deviations) the response to that stimulus was from the average response to all stimuli. This was the case for responses below the average response as well as above. It is shown that the fraction of information carried by the low firing rates of a cell was large--much larger than that carried by the high firing rates. Part of the reason for this is that the probability distribution of different firing rates is biased toward low values (though with fewer very low values than would be predicted by an exponential distribution). Another factor is that the variability of the response is large at intermediate and high firing rates. Another finding is that at short sampling intervals (such as 20 ms) the neurons code information efficiently, by effectively acting as binary variables and behaving less noisily than would be expected of a Poisson process.