Scene Content Research Articles

Adaptive exposure estimation for high dynamic range imaging applied to natural scenes and daylight skies.

Digital imaging of natural scenes and optical phenomena present on them (such as shadows, twilights, and crepuscular rays) can be a very challenging task because of the range spanned by the radiances impinging on the capture system. We propose a novel method for estimating the set of exposure times (bracketing set) needed to capture the full dynamic range of a scene with high dynamic range (HDR) content. The proposed method is adaptive to scene content and to any camera response and configuration, and it works on-line since the exposure times are estimated as the capturing process is ongoing. Besides, it requires no a priori information about scene content or radiance values. The resulting bracketing sets are minimal in the default method settings, but the user can set a tolerance for the maximum percentage of pixel population that is underexposed or saturated, which allows for a higher number of shots if a better signal-to-noise ratio (SNR) in the HDR scene is desired. This method is based on the use of the camera response function that is needed for building the HDR radiance map by stitching together several differently exposed low dynamic range images of the scene. The use of HDR imaging techniques converts our digital camera into a tool for measuring the relative radiance outgoing from each point of the scene, and for each color channel. This is important for accurate characterization of optical phenomena present in the atmosphere while not suffering any loss of information due to its HDR. We have compared our method with the most similar one developed so far [IEEE Trans. Image Process.17, 1864 (2008)]. Results of the experiments carried out for 30 natural scenes show that our proposed method equals or outperforms the previously developed best approach, with less shots and shorter exposure times, thereby asserting the advantage of being adaptive to scene content for exposure time estimation. As we can also tune the balance between capturing time and the SNR in our method, we have compared its SNR performance against that of Barakat's method as well as against a ground-truth HDR image of maximum SNR. Results confirm the success of the proposed method in exploiting its tunability to achieve the desired balance of total Δt and SNR.

Lateralized discrimination of emotional scenes in peripheral vision.

This study investigates whether there is lateralized processing of emotional scenes in the visual periphery, in the absence of eye fixations; and whether this varies with emotional valence (pleasant vs. unpleasant), specific emotional scene content (babies, erotica, human attack, mutilation, etc.), and sex of the viewer. Pairs of emotional (positive or negative) and neutral photographs were presented for 150 ms peripherally (≥6.5° away from fixation). Observers judged on which side the emotional picture was located. Low-level image properties, scene visual saliency, and eye movements were controlled. Results showed that (a) correct identification of the emotional scene exceeded the chance level; (b) performance was more accurate and faster when the emotional scene appeared in the left than in the right visual field; (c) lateralization was equivalent for females and males for pleasant scenes, but was greater for females and unpleasant scenes; and (d) lateralization occurred similarly for different emotional scene categories. These findings reveal discrimination between emotional and neutral scenes, and right brain hemisphere dominance for emotional processing, which is modulated by sex of the viewer and scene valence, and suggest that coarse affective significance can be extracted in peripheral vision.

Social relevance drives viewing behavior independent of low-level salience in rhesus macaques.

Quantifying attention to social stimuli during the viewing of complex social scenes with eye tracking has proven to be a sensitive method in the diagnosis of autism spectrum disorders years before average clinical diagnosis. Rhesus macaques provide an ideal model for understanding the mechanisms underlying social viewing behavior, but to date no comparable behavioral task has been developed for use in monkeys. Using a novel scene-viewing task, we monitored the gaze of three rhesus macaques while they freely viewed well-controlled composed social scenes and analyzed the time spent viewing objects and monkeys. In each of six behavioral sessions, monkeys viewed a set of 90 images (540 unique scenes) with each image presented twice. In two-thirds of the repeated scenes, either a monkey or an object was replaced with a novel item (manipulated scenes). When viewing a repeated scene, monkeys made longer fixations and shorter saccades, shifting from a rapid orienting to global scene contents to a more local analysis of fewer items. In addition to this repetition effect, in manipulated scenes, monkeys demonstrated robust memory by spending more time viewing the replaced items. By analyzing attention to specific scene content, we found that monkeys strongly preferred to view conspecifics and that this was not related to their salience in terms of low-level image features. A model-free analysis of viewing statistics found that monkeys that were viewed earlier and longer had direct gaze and redder sex skin around their face and rump, two important visual social cues. These data provide a quantification of viewing strategy, memory and social preferences in rhesus macaques viewing complex social scenes, and they provide an important baseline with which to compare to the effects of therapeutics aimed at enhancing social cognition.

The great outdoors? Exploring the mental health benefits of natural environments.

There is growing evidence to suggest that exposure to natural environments can be associated with mental health benefits. Proximity to greenspace has been associated with lower levels of stress (Thompson et al., 2012) and reduced symptomology for depression and anxiety (Beyer et al., 2014), while interacting with nature can improve cognition for children with attention deficits (Taylor and Kuo, 2009) and individuals with depression (Berman et al., 2012). A recent epidemiological study has shown that people who move to greener urban areas benefit from sustained improvements in their mental health (Alcock et al., 2014). In this paper we critically review evidence indicating that such mental health benefits are associated with the so-called “restorative” properties of natural environments. In particular we focus on the claim that interaction with (or just passive perception of) natural scene content can be linked to the restoration of limited-capacity attentional resources, in comparison to similar exposure to urban or built scene content.

Volume perception: Disparity extraction and depth representation in complex three-dimensional environments.

Our sensitivity to binocular disparity is exquisite under the best conditions, typically in uncluttered scenes with few small objects. Yet binocular vision can deliver a very strong impression of depth for complex, cluttered scenes with lots of objects and overlaps. How good is disparity processing under these conditions? Here we explored a novel task: depth volume perception, to study how a global representation of depth is obtained using binocular disparity information. We found that the human visual system is sensitive to depth volume but that the volume perceived is dependent on the local and global arrangement of scene content. We also show how a model of early disparity extraction and combination can account for some of the biases found. Our work shows that the visual system is not able to correctly represent and interpret disparity for all locations in a complex three-dimensional scene.

Real‐Time Isosurface Extraction With View‐Dependent Level of Detail and Applications

AbstractVolumetric scalar data sets are common in many scientific, engineering and medical applications where they originate from measurements or simulations. Furthermore, they can represent geometric scene content, e.g. as distance or density fields. Often isosurfaces are extracted, either for indirect volume visualization in the former category, or to simply obtain a polygonal representation in case of the latter. However, even moderately sized volume data sets can result in complex isosurfaces which are challenging to recompute in real time, e.g. when the user modifies the isovalue or when the data itself are dynamic. In this paper, we present a GPU‐friendly algorithm for the extraction of isosurfaces, which provides adaptive level of detail rendering with view‐dependent tessellation. It is based on a longest edge bisection scheme where the resulting tetrahedral cells are subdivided into four hexahedra, which then form the domain for the subsequent isosurface extraction step. Our algorithm generates meshes with good triangle quality even for highly non‐linear scalar data. In contrast to previous methods, it does not require any stitching between regions of different levels of detail. As all computation is performed at run time and no pre‐processing is required, the algorithm naturally supports dynamic data and allows us to change isovalues at any time.

Common source identification of images in large databases

Photo-response non-uniformity noise patterns are a robust way to identify the source of an image. However, identifying a common source of images in a large database may be impractical due to long computation times. In this paper a solution for large volume digital camera identification is proposed, which combines, and sometimes slightly modifies, existing methods for a 500 times improvement in the speed of common source identification. Single image comparisons are often plagued by considerable noise contamination from scene content and random noise, which makes it harder to accomplish reliable common source identification. Therefore a new method is introduced that can increase true positive rates by more than 45% at very low computation costs. Analysis of real data from a fraud case shows the effectiveness of the proposed method. As a whole the proposed solution makes it possible to analyze a large database in forensically relevant time, without resorting to large and expensive computer clusters.

Modelling fast forms of visual neural plasticity using a modified second-order motion energy model.

The Adelson-Bergen motion energy sensor is well established as the leading model of low-level visual motion sensing in human vision. However, the standard model cannot predict adaptation effects in motion perception. A previous paper Pavan et al.(Journal of Vision 10:1-17, 2013) presented an extension to the model which uses a first-order RC gain-control circuit (leaky integrator) to implement adaptation effects which can span many seconds, and showed that the extended model's output is consistent with psychophysical data on the classic motion after-effect. Recent psychophysical research has reported adaptation over much shorter time periods, spanning just a few hundred milliseconds. The present paper further extends the sensor model to implement rapid adaptation, by adding a second-order RC circuit which causes the sensor to require a finite amount of time to react to a sudden change in stimulation. The output of the new sensor accounts accurately for psychophysical data on rapid forms of facilitation (rapid visual motion priming, rVMP) and suppression (rapid motion after-effect, rMAE). Changes in natural scene content occur over multiple time scales, and multi-stage leaky integrators of the kind proposed here offer a computational scheme for modelling adaptation over multiple time scales.

The timing and directional connectivity of human frontoparietal and ventral visual attention networks in emotional scene perception

Electrocortical and hemodynamic measures reliably identify enhanced activity in the ventral and dorsal visual cortices during the perception of emotionally arousing versus neutral images, an effect that may reflect directive feedback from the subcortical amygdala. However, other brain regions strongly modulate visual attention, such as frontal eye fields (FEF) and intraparietal sulcus (IPS). Here we employ rapid sampling of BOLD signal (4Hz) in the amygdala, fusiform gyrus (FG), FEF and IPS in 42 human participants as they viewed a series of emotional and neutral natural scene photographs balanced for luminosity and complexity, to test whether emotional discrimination is evident in dorsal structures prior to such discrimination in the amygdala and FG. Granger causality analyses were used to assess directional connectivity within dorsal and ventral networks. Results demonstrate emotionally-enhanced peak BOLD signal in the amygdala, FG, FEF, and IPS, with the onset of BOLD signal discrimination occurring between 2 and 3s after stimulus onset in ventral structures, and between 4 and 5s in FEF and IPS. Granger causality estimates yield stronger directional connectivity from IPS to FEF than the reverse in this emotional picture paradigm. Consistent with a reentrant perspective of emotional scene perception, greater directional connectivity was found from the amygdala to FG compared to the reverse. These data support a perspective in which the registration of emotional scene content is orchestrated by the amygdala and rostral inferotemporal visual cortex.

Startle modulation during emotional anticipation and perception.

The startle reflex is potentiated when anticipating emotional, compared to neutral, pictures. This study investigated the time course of reflex modulation during anticipation and the impact of informative cuing on picture perception. Colors were used to signal the thematic content of emotional and neutral scenes; blink response modulation was measured by presenting acoustic startle probes 3, 2, or 1 s before picture onset or 2 s after picture onset. During anticipation of neutral scenes, blink magnitude showed increasing attenuation as picture onset approached, consistent with a modality-directed vigilance account. Conversely, when anticipating emotional scenes, reflex magnitude did not change over time, and blinks elicited closest to picture onset were potentiated compared to neutral. During perception, the expected reflex potentiation for unpleasant pictures was not found, suggesting that cuing may dampen defensive activation.

A Statistical Approach to Automatically Detect How Many Persons Appear in a Video

Face indexing is a very popular research topic and it has been investigated over the last 10 years. It can be used for a wide range of applications such as automatic video content analysis, data mining, video annotation and labeling, etc. In this work a statistical approach to address this challenging issue is presented: the number of persons that are present in a generic video (even having low resolution and/or taken from a mobile camera) is automatically detected and also the intervals of frames in which each person appears are extracted. The main contributions of the proposed work are that no initializations neither a priory knowledge about the scene contents are required. Moreover, this approach introduces a generalized version of the k-means method that, through different statistical indices, automatically determines the number of people in the scene.

Nonrigid registration with free-form deformation model of multilevel uniform cubic B-splines: application to image registration and distortion correction of spectral image cubes.

In spectral imaging, spatial and spectral information of an image scene are combined. There exist several technologies that allow the acquisition of this kind of data. Depending on the optical components used in the spectral imaging systems, misalignment between image channels can occur. Further, the projection of some systems deviates from that of a perfect optical lens system enough that a distortion of scene content in the images becomes apparent to the observer. Correcting distortion and misalignment can be complicated for spectral image data if they are different at each image channel. In this work, we propose an image registration and distortion correction scheme for spectral image cubes that is based on a free-form deformation model of uniform cubic B-splines with multilevel grid refinement. This scheme is adaptive with respect to image size, degree of misalignment, and degree of distortion, and in that sense is superior to previous approaches. We support our proposed scheme with empirical data from a Bragg-grating-based hyperspectral imager, for which a registration accuracy of approximately one pixel was achieved.

A Data-Driven Stochastic Approach for Unmixing Hyperspectral Imagery

In this paper, we propose a two-step Bayesian approach to handle the ill-posed nature of the unmixing problem for accurately estimating the abundances. The abundances are dependent on the scene contents and they represent mixing proportions of the endmembers over an area. In this work, a linear mixing model (LMM) is used for the image formation process in order to derive the data term. In the first step, a Huber-Markov random field (HMRF)-based prior distribution is assumed to model the dependencies within the abundances across the spectral space of the data. The threshold used in the HMRF prior is derived from an initial estimate of abundances obtained using the matched filters. This makes the HMRF prior data-driven, i.e., dHMRF. Final abundance maps are obtained in the second step within a maximum a posteriori probability (MAP) framework, and the objective function is optimized using the particle swarm optimization (PSO). Theoretical analysis is carried out to show the effectiveness of the proposed method. The approach is evaluated using the synthetic and real AVIRIS Cuprite data. The proposed method has the following advantages. 1) The estimated abundances are resistant to noise since they are based on an initial estimate that has high signal-to-noise ratio (SNR). 2) The variance in the abundance maps is well preserved since the threshold in the dHMRF is derived from the data.

A game attention model for efficient bit rate allocation in cloud gaming

The widespread availability of broadband internet access and the growth in server-based processing have provided an opportunity to run games away from the player into the cloud and offer a new promising service known as cloud gaming. The concept of cloud gaming is to render a game in the cloud and stream the resulting game scenes to the player as a video sequence over a broadband connection. To meet the stringent network bandwidth requirements of cloud gaming and support more players, efficient bit rate reduction techniques are needed. In this paper, we introduce the concept of game attention model (GAM), which is basically a game context-based visual attention model, as a means for reducing the bit rate of the streaming video more efficiently. GAM estimates the importance of each macro-block in a game frame from the player's perspective and allows encoding the less important macro-blocks with lower bit rate. We have evaluated nine game video sequences, covering a wide range of game genre and a spectrum of scene content in terms of details, motion and brightness. Our subjective assessment shows that by integrating this model into the cloud gaming framework, it is possible to decrease the required bit rate by nearly 25 % on average, while maintaining a relatively high user quality of experience. This clearly enables players with limited communication resources to benefit from cloud gaming with acceptable quality.

Moving object recognition under simulated prosthetic vision using background-subtraction-based image processing strategies

A visual prosthesis that applies electrical stimulation to different parts of the visual pathway has been proposed as a viable approach to restore functional vision. However, the created percept is currently limited due to the low-resolution images elicited from a limited number of stimulating electrodes. Thus, methods to optimize the visual percepts providing useful visual information are being considered. We used two image-processing strategies based on a novel background subtraction technique to optimize the content of dynamic scenes of daily life. Psychophysical results showed that background reduction, or background reduction with foreground enhancement, increased response accuracy compared with methods that directly merged pixels to lower resolution. By adding more gray scale information, a background reduction/foreground enhancement strategy resulted in the best performance and highest recognition accuracy. Further development of image-processing modules for a visual prosthesis based on these results will assist implant recipients to avoid dangerous situations and attain independent mobility in daily life.

Content-adaptive high-resolution hyperspectral video acquisition with a hybrid camera system

We present a hybrid camera system that combines optical designs with computational processing to achieve content-adaptive high-resolution hyperspectral video acquisition. In particular, we record two video streams: one high-spatial resolution RGB video and one low-spatial resolution hyperspectral video in which the recorded points are dynamically selected using a spatial light modulator (SLM). Then through video-frame registration and a spatio-temporal spreading of the co-located spectral/RGB information, video with high spatial and spectral resolution is produced. The sampling patterns on the SLM are generated on-the-fly according to the scene content, which fully exploits the self-adaptivity of the hybrid camera system. With an experimental prototype, we demonstrate significantly improved accuracy and efficiency as compared to the state-of-the-art.

Unearthing the picturesque: The validity of the preference matrix as a measure of landscape aesthetics

Previous research has provided inconclusive support for the preference matrix; a psychological framework of landscape aesthetics rooted in a long tradition of studying scenic quality. Given recent insights into the important implications that aesthetics have for health and well-being, the aim of the present study was to re-assess the validity of the preference matrix following the implementation of a series of methodological improvements. These entailed: a set of item definitions piloted for high comprehension; adequate statistical control for confounding influences by scene content and user experience; a substantially sized, highly varied image database; and the employment of statistical instruments to formally test for nonlinear relationships. An ordinal mixed effects model provides convergent evidence for our hypothesis that each of predictors in the preference matrix is independently predictive of scene aesthetics. In addition, we find support for an interaction between the constructs of Coherence and Complexity and show that levels of both natural and built character, as well as familiarity, are uniquely predictive of scene attractiveness. The present findings underline the role of the preference matrix as a potential tool in informing evidence-based design.

The conceptual metaphors of narrative structure: Gestural evidence for spatialized form in storytelling

This paper explores conceptual tools whereby narratively competent adults conceptualize the structure of literary events, as opposed to their scene content. My focus lies on how narrativity as a mode of thought is constituted through metanarrative discourse and what role embodied representations play in it. This global level of story cognition takes the form of conceptual metaphors such as TIME IS A PATH, CAUSALITY IS FORCE, or THEMATIC REALMS ARE SPACES/PLANES. Two kinds of evidence for this claim are combined: (a) linguistic metaphors for story gist, and, more extensively, (b) metaphorical gestures that accompany story summarization and commentary. Based on footage in which German literary critics discuss books, my specific task is to identify the various dimensions of story logic that gestures refer to. Overall, the data suggests that narrative form is systematically rooted in spatial logic and that dedicated structural devices dynamically co-evolve with the retelling of content. The study thus contributes a demonstration of Lakoff’s (1987) “spatialization of form hypothesis”, i.e. the wide ranging claim that structural cognition is rooted in image schemas.

Abnormal Scene Change Detection from a Moving Camera Using Bags of Patches and Spider-Web Map

This paper proposes a novel surveillance system for detecting exceptional scene changes as abnormal events with a mobile camera mounted on a robot. In contrast to abnormal event analysis using fixed cameras, three key problems should be tackled in this system, i.e., scene construction, robot localization, and scene comparison. For the first problem, scene construction, a clustering scheme is proposed for extracting a set of key frames from the surveillance environment. Each key frame is further divided into a set of patches, which forms a sparse representation for representing scene contents. In addition to the compression effect, the scheme can tackle the effects of misalignment and lighting changes well. For the localization problem, a novel patch matching method is proposed to reduce not only the size of the search space but also the size of the feature dimensions in similarity matching. To prune the search space, a set of projection kernels is used to construct a ring structure. Then, one order of time complexity in the similarity calculation can be reduced from the structure. After scene searching, the robot location is not always guaranteed to be successfully registered to the scene map. Thus, a novel spider-web map is proposed to tackle the effect of misalignment and then detect different exceptional scene changes from the videos. The proposed method has been rigorously tested on a variety of videos to demonstrate its superiority in object detection and abnormal scene change detection.

Interocular correlation between human retinal images at six-weeks of age

Introduction: The developing visual system must make an appropriate vergence movement to fixate a target and permit binocular function. The visual system must plan vergence responses based on retinal images complicated by external factors (e.g. variable scene content) and developmental hurdles (e.g. immature ocular optics). Our goal was to determine the uniqueness of vergence solutions given variation in both scene structure and the optics of the developing eye.