Scene Content Research Articles

Fractal dimension and the navigational information provided by natural scenes.

Recent work on virtual reality navigation in humans has suggested that navigational success is inversely correlated with the fractal dimension (FD) of artificial scenes. Here we investigate the generality of this claim by analysing the relationship between the fractal dimension of natural insect navigation environments and a quantitative measure of the navigational information content of natural scenes. We show that the fractal dimension of natural scenes is in general inversely proportional to the information they provide to navigating agents on heading direction as measured by the rotational image difference function (rotIDF). The rotIDF determines the precision and accuracy with which the orientation of a reference image can be recovered or maintained and the range over which a gradient descent in image differences will find the minimum of the rotIDF, that is the reference orientation. However, scenes with similar fractal dimension can differ significantly in the depth of the rotIDF, because FD does not discriminate between the orientations of edges, while the rotIDF is mainly affected by edge orientation parallel to the axis of rotation. We present a new equation for the rotIDF relating navigational information to quantifiable image properties such as contrast to show (1) that for any given scene the maximum value of the rotIDF (its depth) is proportional to pixel variance and (2) that FD is inversely proportional to pixel variance. This contrast dependence, together with scene differences in orientation statistics, explains why there is no strict relationship between FD and navigational information. Our experimental data and their numerical analysis corroborate these results.

The influence of behavioral relevance on the processing of global scene properties: An ERP study

Recent work studying the temporal dynamics of visual scene processing (Harel et al., 2016) has found that global scene properties (GSPs) modulate the amplitude of early Event-Related Potentials (ERPs). It is still not clear, however, to what extent the processing of these GSPs is influenced by their behavioral relevance, determined by the goals of the observer. To address this question, we investigated how behavioral relevance, operationalized by the task context impacts the electrophysiological responses to GSPs. In a set of two experiments we recorded ERPs while participants viewed images of real-world scenes, varying along two GSPs, naturalness (manmade/natural) and spatial expanse (open/closed). In Experiment 1, very little attention to scene content was required as participants viewed the scenes while performing an orthogonal fixation-cross task. In Experiment 2 participants saw the same scenes but now had to actively categorize them, based either on their naturalness or spatial expense. We found that task context had very little impact on the early ERP responses to the naturalness and spatial expanse of the scenes: P1, N1, and P2 could distinguish between open and closed scenes and between manmade and natural scenes across both experiments. Further, the specific effects of naturalness and spatial expanse on the ERP components were largely unaffected by their relevance for the task. A task effect was found at the N1 and P2 level, but this effect was manifest across all scene dimensions, indicating a general effect rather than an interaction between task context and GSPs. Together, these findings suggest that the extraction of global scene information reflected in the early ERP components is rapid and very little influenced by top-down observer-based goals.

Watch to Edit: Video Retargeting using Gaze

AbstractWe present a novel approach to optimally retarget videos for varied displays with differing aspect ratios by preserving salient scene content discovered via eye tracking. Our algorithm performs editing with cut, pan and zoom operations by optimizing the path of a cropping window within the original video while seeking to (i) preserve salient regions, and (ii) adhere to the principles of cinematography. Our approach is (a) content agnostic as the same methodology is employed to re‐edit a wide‐angle video recording or a close‐up movie sequence captured with a static or moving camera, and (b) independent of video length and can in principle re‐edit an entire movie in one shot.Our algorithm consists of two steps. The first step employs gaze transition cues to detect time stamps where new cuts are to be introduced in the original video via dynamic programming. A subsequent step optimizes the cropping window path (to create pan and zoom effects), while accounting for the original and new cuts. The cropping window path is designed to include maximum gaze information, and is composed of piecewise constant, linear and parabolic segments. It is obtained via L(1) regularized convex optimization which ensures a smooth viewing experience. We test our approach on a wide variety of videos and demonstrate significant improvement over the state‐of‐the‐art, both in terms of computational complexity and qualitative aspects. A study performed with 16 users confirms that our approach results in a superior viewing experience as compared to gaze driven re‐editing [JSSH15] and letterboxing methods, especially for wide‐angle static camera recordings.

Scene Semantic Recognition Based on Probability Topic Model

In recent years, scene semantic recognition has become the most exciting and fastest growing research topic. Lots of scene semantic analysis methods thus have been proposed for better scene content interpretation. By using latent Dirichlet allocation (LDA) to deduce the effective topic features, the accuracy of image semantic recognition has been significantly improved. Besides, the method of extracting deep features by layer-by-layer iterative computation using convolutional neural networks (CNNs) has achieved great success in image recognition. The paper proposes a method called DF-LDA, which is a hybrid supervised–unsupervised method combined CNNs with LDA to extract image topics. This method uses CNNs to explore visual features that are more suitable for scene images, and group the features of salient semantics into visual topics through topic models. In contrast to the LDA as a tool for simply extracting image semantics, our approach achieves better performance on three datasets that contain various scene categories.

Gaze patterns reveal how situation models and text representations contribute to episodic text memory

When recalling something you have previously read, to what degree will such episodic remembering activate a situation model of described events versus a memory representation of the text itself? The present study was designed to address this question by recording eye movements of participants who recalled previously read texts while looking at a blank screen. An accumulating body of research has demonstrated that spontaneous eye movements occur during episodic memory retrieval and that fixation locations from such gaze patterns to a large degree overlap with the visuospatial layout of the recalled information. Here we used this phenomenon to investigate to what degree participants’ gaze patterns corresponded with the visuospatial configuration of the text itself versus a visuospatial configuration described in it. The texts to be recalled were scene descriptions, where the spatial configuration of the scene content was manipulated to be either congruent or incongruent with the spatial configuration of the text itself. Results show that participants’ gaze patterns were more likely to correspond with a visuospatial representation of the described scene than with a visuospatial representation of the text itself, but also that the contribution of those representations of space is sensitive to the text content. This is the first demonstration that eye movements can be used to discriminate on which representational level texts are remembered and the findings provide novel insight into the underlying dynamics in play.

Calculating Retinal Contrast from Scene Content: A Program.

This paper describes a computer program for calculating the contrast image on the human retina from an array of scene luminances. We used achromatic transparency targets and measured test target's luminances with meters. We used the CIE standard Glare Spread Function (GSF) to calculate the array of retinal contrast. This paper describes the CIE standard, the calculation and the analysis techniques comparing the calculated retinal image with observer data. The paper also describes in detail the techniques of accurate measurements of HDR scenes, conversion of measurements to input data arrays, calculation of the retinal image, including open source MATLAB code, pseudocolor visualization of HDR images that exceed the range of standard displays, and comparison of observed sensations with retinal stimuli.

Performance Characterization of Image Feature Detectors in Relation to the Scene Content Utilizing a Large Image Database

Selecting the most suitable local invariant feature detector for a particular application has rendered the task of evaluating feature detectors a critical issue in vision research. Although the literature, offers a variety of comparison works focusing on performance evaluation of image feature detectors under several types of image transformations, the influence of the scene content on the performance of local feature detectors has received little attention so far. This paper, aims to bridge this gap with a new framework for determining the type of scenes which maximize and minimize the performance of detectors in terms of repeatability rate. The results are presented for several state-of-the-art feature detectors that have been obtained using a large image database of 20482 images under JPEG compression, uniform light and blur changes with 539 different scenes captured from real-world scenarios. These results provide new insights into the behavior of feature detectors.

An Adaptive Quad-Tree Depth Range Prediction Mechanism for HEVC

A next-generation video coding standard High Efficiency Video Coding (HEVC) provides higher video quality and lower compression bit rate but leads to very high encoding complexity, especially in the quad-tree-based coding tree unit partitioning process. To reduce the computational complexity of HEVC, in this paper, we propose an adaptive quad-tree depth range prediction mechanism. First, the proposed mechanism defines the similar region flag to distinguish between the similar region and the non-similar region. Then, two algorithms, the similar region depth range prediction algorithm and the non-similar region depth range prediction algorithm, are proposed. The similar region depth range prediction algorithm estimates the features of the similar region based on the coding unit depth of this region. The optimal depth of this region can be predicted. The non-similar region depth range prediction algorithm can skip low probability tree nodes based on the depth correlation coefficient, which is calculated based on scene content change. Both the similar region depth range prediction algorithm and the non-similar region depth range prediction algorithm show more than 90% predictive accuracy. Experimental results show that under random access configuration and low delay configuration, the proposed mechanism can yield 28.17% and 32.99% computational complexity reduction with negligible rate distortion performance loss, respectively, compared with HM16.9. The results show that the proposed mechanism is expected to be applied in real-time environments.

Scene content is predominantly conveyed by high spatial frequencies in scene-selective visual cortex.

In complex real-world scenes, image content is conveyed by a large collection of intertwined visual features. The visual system disentangles these features in order to extract information about image content. Here, we investigate the role of one integral component: the content of spatial frequencies in an image. Specifically, we measure the amount of image content carried by low versus high spatial frequencies for the representation of real-world scenes in scene-selective regions of human visual cortex. To this end, we attempted to decode scene categories from the brain activity patterns of participants viewing scene images that contained the full spatial frequency spectrum, only low spatial frequencies, or only high spatial frequencies, all carefully controlled for contrast and luminance. Contrary to the findings from numerous behavioral studies and computational models that have highlighted how low spatial frequencies preferentially encode image content, decoding of scene categories from the scene-selective brain regions, including the parahippocampal place area (PPA), was significantly more accurate for high than low spatial frequency images. In fact, decoding accuracy was just as high for high spatial frequency images as for images containing the full spatial frequency spectrum in scene-selective areas PPA, RSC, OPA and object selective area LOC. We also found an interesting dissociation between the posterior and anterior subdivisions of PPA: categories were decodable from both high and low spatial frequency scenes in posterior PPA but only from high spatial frequency scenes in anterior PPA; and spatial frequency was explicitly decodable from posterior but not anterior PPA. Our results are consistent with recent findings that line drawings, which consist almost entirely of high spatial frequencies, elicit a neural representation of scene categories that is equivalent to that of full-spectrum color photographs. Collectively, these findings demonstrate the importance of high spatial frequencies for conveying the content of complex real-world scenes.

DiffuserCam: lensless single-exposure 3D imaging

We demonstrate a compact, easy-to-build computational camera for single-shot three-dimensional (3D) imaging. Our lensless system consists solely of a diffuser placed in front of an image sensor. Every point within the volumetric field-of-view projects a unique pseudorandom pattern of caustics on the sensor. By using a physical approximation and simple calibration scheme, we solve the large-scale inverse problem in a computationally efficient way. The caustic patterns enable compressed sensing, which exploits sparsity in the sample to solve for more 3D voxels than pixels on the 2D sensor. Our 3D reconstruction grid is chosen to match the experimentally measured two-point optical resolution, resulting in 100 million voxels being reconstructed from a single 1.3 megapixel image. However, the effective resolution varies significantly with scene content. Because this effect is common to a wide range of computational cameras, we provide a new theory for analyzing resolution in such systems.

A Developmental Approach to Machine Learning?

Visual learning depends on both the algorithms and the training material. This essay considers the natural statistics of infant- and toddler-egocentric vision. These natural training sets for human visual object recognition are very different from the training data fed into machine vision systems. Rather than equal experiences with all kinds of things, toddlers experience extremely skewed distributions with many repeated occurrences of a very few things. And though highly variable when considered as a whole, individual views of things are experienced in a specific order – with slow, smooth visual changes moment-to-moment, and developmentally ordered transitions in scene content. We propose that the skewed, ordered, biased visual experiences of infants and toddlers are the training data that allow human learners to develop a way to recognize everything, both the pervasively present entities and the rarely encountered ones. The joint consideration of real-world statistics for learning by researchers of human and machine learning seems likely to bring advances in both disciplines.

Road Scene Content Analysis for Driver Assistance and Autonomous Driving

This paper aims to develop a vision-based driver assistance system for scene awareness using video frames obtained from a dashboard camera. A saliency image map is devised with features pertinent to the driving scene. This saliency map mimics the human contour and motion sensitive visual perception by extracting spatial, spectral, and temporal information from the input frames and applying entropy driven image-context-feature data fusion. The resultant fusion output comprises high-level descriptors for still segment boundaries and non-stationary object appearance. Following the segmentation and foreground object detection stage, an adaptive maximum likelihood classifier selects road surface regions. The proposed scene driven vision system improves the driver’s situational awareness by enabling adaptive road surface classification. As experimental results demonstrate, context-aware low-level to high-level information fusion based on human vision model produces superior segmentation, tracking, and classification results that lead to high- level abstraction of driving scene.

Genetic Influence on Eye Movements to Complex Scenes at Short Timescales

Where one looks within their environment constrains one's visual experiences, directly affects cognitive, emotional, and social processing [1-4], influences learning opportunities [5], and ultimately shapes one's developmental path. While there is a high degree of similarity across individuals with regard to which features of a scene are fixated [6-8], large individual differences are also present, especially in disorders of development [9-13], and clarifying the origins of these differences is essential to understand the processes by which individuals develop within the complex environments in which they exist and interact. Toward this end, a recent paper [14] found that "social visual engagement"-namely, gaze to eyes and mouths of faces-is strongly influenced by genetic factors. However, whether genetic factors influence gaze to complex visual scenes more broadly, impacting how both social and non-social scene content are fixated, as well as general visual exploration strategies, has yet to be determined. Using a behavioral genetic approach and eye tracking data from a large sample of 11-year-old human twins (233 same-sex twin pairs; 51% monozygotic, 49% dizygotic), we demonstrate that genetic factors do indeed contribute strongly to eye movement patterns, influencing both one's general tendency for visual exploration of scene content, as well as the precise moment-to-moment spatiotemporal pattern of fixations during viewing of complex social and non-social scenes alike. This study adds to a now growing set of results that together illustrate how genetics may broadly influence the process by which individuals actively shape and create their own visual experiences.

An illustrated prehistory of the Jubbah oasis: Reconstructing Holocene occupation patterns in north‐western Saudi Arabia from rock art and inscriptions

AbstractA systematic survey of rock art and associated archaeological features in the Jubbah oasis provides evidence of Holocene occupation from the early Holocene to the present. In total 1249 panels with rock art and inscriptions, and 159 archaeological sites, were recorded on twelve different jebels. Analyses of rock art content and engraving stratigraphy indicate that the iconic Jubbah style had a long tradition among pre‐pastoral hunters and continued to be used by early herders. We also identify a distinct body of rock art that pre‐dates the Jubbah style and may be associated with a nearby Epipalaeolithic site. Our systematic dataset identifies a body of Bronze Age rock art that is further supported by the material culture and radiocarbon dates obtained from the remains of disturbed cairns. The rock art in Jubbah appears to have been created throughout the Holocene occupation of the oasis and similarities in the representation of animals, choice of location and content of rock art scenes are evident from the Bronze Age to the early modern period, and perhaps into the twentieth century. Moreover, rock art and epigraphy suggest that occupation phases in Jubbah were sustained long enough for the repeated development of unique local characteristics throughout the Holocene.

Learning scene-aware image priors with high-order Markov random fields

Many methods have been proposed to learn image priors from natural images for the ill-posed image restoration tasks. However, many prior learning algorithms assume that a general prior distribution is suitable for over all kinds of images. Since the contents of the natural images and the corresponding low-level statistical characteristics vary from scene to scene, we argue that learning a universal generative prior for all natural images may be imperfect. Although the universal generative prior can remove artifacts and reserve a natural smoothness in image restoration, it also tends to introduce unreal flatness and clutter textures. To address this issue, in this paper, we present to learn a scene-aware image prior based on the high-order Markov random field (MRF) model (SA-MRF). With this model, we jointly learn a set of shared low-level features and different potentials for specific scene contents. In prediction, a good prior can be adapted to the given degenerated image with the scene content perception. Experimental results on the image denoising and inpainting tasks demonstrate the efficiency of the SA-MRF on both numerical evaluation and visual compression.

Group‐Theme Recoloring for Multi‐Image Color Consistency

AbstractModifying the colors of an image is a fundamental editing task with a wide range of methods available. Manipulating multiple images to share similar colors is more challenging, with limited tools available. Methods such as color transfer are effective in making an image share similar colors with a target image; however, color transfer is not suitable for modifying multiple images. Approaches for color consistency for photo collections give good results when the photo collection contains similar scene content, but are not applicable for general input images. To address these gaps, we propose an application framework for achieving color consistency for multi‐image input. Our framework derives a group color theme from the input images′ individual color palettes and uses this group color theme to recolor the image collection. This group‐theme recoloring provides an effective way to ensure color consistency among multiple images and naturally lends itself to the inclusion of an additional external color theme. We detail our group‐theme recoloring approach and demonstrate its effectiveness on a number of examples.

Imaging spectrometer stray spectral response: In-flight characterization, correction, and validation

We present a new method for more accurate in-flight calibration and correction of imaging spectrometer spectral response functions. Non-Gaussian tails of spectral response functions can be difficult to characterize in the laboratory, and calibration can shift during deployment. Consequently, in-flight techniques are useful for validating and updating laboratory measurements. Our approach exploits predictable changes in the shape of the oxygen A band across varying surface elevation, with diverse scene content providing numerical leverage to characterize spectral response tails 3–4 orders of magnitude below the peak. We present a correction to recover the nominal response function, and show case studies based on NASA's Next Generation Airborne Visible Infrared Imaging Spectrometer (AVIRIS-NG). Corrected radiances are better conditioned for downstream analysis by sensitive atmospheric codes. We evaluate accuracy using multiple independent standards: simulation studies; consistency with laboratory measurements; elimination of a surface pressure retrieval bias; better alignment of retrieved reflectance with ground reference data; and statistics of over 250 flightlines from a campaign across the Indian Subcontinent showing consistent improvements in atmospheric correction.

Auto white balance by surface reflection decomposition.

Auto white balance (AWB) is an important operation in color imaging applications. Most existing AWB algorithms rely on some physical features and statistical properties of natural scenes. However, the AWB algorithms using statistical properties are sensitive to the statistics of the scene contents. Therefore, it is highly desirable to find physical features that are more robust and relatively insensitive to scene contents. In this paper, we propose such physical features based on surface reflection decomposition. Light reflection from most object surfaces can be decomposed into a specular component and a diffuse component. Instead of trying to find the common axis of the color planes as in past algorithms, we estimate the illuminant chromaticity by searching through the light source candidates to find the one that will best cancel the specular components. We provide two formulations: the minimum projected area algorithm and the minimum total variation algorithm for estimation of the scene-illuminant chromaticity. Both show very favorable results compared with other published algorithms.

Validity and reliability of naturalistic driving scene categorization Judgments from crowdsourcing

A common challenge with processing naturalistic driving data is that humans may need to categorize great volumes of recorded visual information. By means of the online platform CrowdFlower, we investigated the potential of crowdsourcing to categorize driving scene features (i.e., presence of other road users, straight road segments, etc.) at greater scale than a single person or a small team of researchers would be capable of. In total, 200 workers from 46 different countries participated in 1.5days. Validity and reliability were examined, both with and without embedding researcher generated control questions via the CrowdFlower mechanism known as Gold Test Questions (GTQs).By employing GTQs, we found significantly more valid (accurate) and reliable (consistent) identification of driving scene items from external workers. Specifically, at a small scale CrowdFlower Job of 48 three-second video segments, an accuracy (i.e., relative to the ratings of a confederate researcher) of 91% on items was found with GTQs compared to 78% without. A difference in bias was found, where without GTQs, external workers returned more false positives than with GTQs. At a larger scale CrowdFlower Job making exclusive use of GTQs, 12,862 three-second video segments were released for annotation. Infeasible (and self-defeating) to check the accuracy of each at this scale, a random subset of 1012 categorizations was validated and returned similar levels of accuracy (95%).In the small scale Job, where full video segments were repeated in triplicate, the percentage of unanimous agreement on the items was found significantly more consistent when using GTQs (90%) than without them (65%). Additionally, in the larger scale Job (where a single second of a video segment was overlapped by ratings of three sequentially neighboring segments), a mean unanimity of 94% was obtained with validated-as-correct ratings and 91% with non-validated ratings. Because the video segments overlapped in full for the small scale Job, and in part for the larger scale Job, it should be noted that such reliability reported here may not be directly comparable. Nonetheless, such results are both indicative of high levels of obtained rating reliability.Overall, our results provide compelling evidence for CrowdFlower, via use of GTQs, being able to yield more accurate and consistent crowdsourced categorizations of naturalistic driving scene contents than when used without such a control mechanism. Such annotations in such short periods of time present a potentially powerful resource in driving research and driving automation development.

A local variance based approach to alleviate the scene content interference for source camera identification

Identifying the source camera of images is becoming increasingly important nowadays. A popular approach is to use a type of pattern noise called photo-response non-uniformity (PRNU). The noise of image contains the patterns which can be used as a fingerprint. Despite that, the PRNU-based approach is sensitive towards scene content and image intensity. The identification is poor in areas having low or saturated intensity, or in areas with complicated texture. The reliability of different regions is difficult to model in that it depends on the interaction of scene content and the characteristics of the denoising filter used to extract the noise. In this paper, we showed that the local variance of the noise residual can measure the reliability of the pixel for PRNU-based source camera identification. Hence, we proposed to use local variance to characterize the severeness of the scene content artifacts. The local variance is then incorporated to the general matched filter and peak to correlation energy (PCE) detector to provide an optimal framework for signal detection. The proposed method is tested against several state-of-art methods. The experimental results show that the local variance based approach outperformed other state-of-the-art methods in terms of identification accuracy.