Scene Content Research Articles

Full-Scale Assessment of Pansharpening Through Polynomial Fitting of Multiscale Measurements

Pansharpening techniques aim at improving the spatial resolution of a multispectral data set (MS) by using a panchromatic image (Pan) acquired on the same scene with a greater spatial resolution and consequently a lower ground sample distance (GSD). Usually, a quantitative assessment of the fused products cannot be directly performed because of the lack of a reference MS data set with the same GSD of the Pan. A well-known solution is Wald's protocol: The original MS and Pan are spatially degraded, the reducing factor being the ratio between their GSDs. Pansharpening is then performed between the reduced MS and Pan data sets, and the fused products are compared with the original MS, which can be used as reference. In this protocol, fusion performances are assumed to be independent of the scale so that the results at the reduced scale are an estimation of those at the original resolution. This hypothesis can be more or less reliable, depending on the sensor and/or the scene content. The objective of this paper is to propose a new methodology to infer the unknown performances of a pansharpening method at full scale. For this purpose, multiple sets of fused images are computed at degraded scales by downsampling Pan and MS data sets by means of the sensor modulation transfer function. Multiscale quality/distortion measurements are fitted by linear and quadratic polynomials in order to extrapolate their full-scale values. Once the proposed protocol has been assessed in the presence of reference originals, the obtained results are extended to the case where the reference image is not available.

How do field of view and resolution affect the information content of panoramic scenes for visual navigation? A computational investigation.

The visual systems of animals have to provide information to guide behaviour and the informational requirements of an animal’s behavioural repertoire are often reflected in its sensory system. For insects, this is often evident in the optical array of the compound eye. One behaviour that insects share with many animals is the use of learnt visual information for navigation. As ants are expert visual navigators it may be that their vision is optimised for navigation. Here we take a computational approach in asking how the details of the optical array influence the informational content of scenes used in simple view matching strategies for orientation. We find that robust orientation is best achieved with low-resolution visual information and a large field of view, similar to the optical properties seen for many ant species. A lower resolution allows for a trade-off between specificity and generalisation for stored views. Additionally, our simulations show that orientation performance increases if different portions of the visual field are considered as discrete visual sensors, each giving an independent directional estimate. This suggests that ants might benefit by processing information from their two eyes independently.Electronic supplementary materialThe online version of this article (doi:10.1007/s00359-015-1052-1) contains supplementary material, which is available to authorized users.

Emotional facial expressions evoke faster orienting responses, but weaker emotional responses at neural and behavioural levels compared to scenes: A simultaneous EEG and facial EMG study

In the current study, electroencephalography (EEG) was recorded simultaneously with facial electromyography (fEMG) to determine whether emotional faces and emotional scenes are processed differently at the neural level. In addition, it was investigated whether these differences can be observed at the behavioural level via spontaneous facial muscle activity. Emotional content of the stimuli did not affect early P1 activity. Emotional faces elicited enhanced amplitudes of the face-sensitive N170 component, while its counterpart, the scene-related N100, was not sensitive to emotional content of scenes. At 220–280ms, the early posterior negativity (EPN) was enhanced only slightly for fearful as compared to neutral or happy faces. However, its amplitudes were significantly enhanced during processing of scenes with positive content, particularly over the right hemisphere. Scenes of positive content also elicited enhanced spontaneous zygomatic activity from 500–750ms onwards, while happy faces elicited no such changes. Contrastingly, both fearful faces and negative scenes elicited enhanced spontaneous corrugator activity at 500–750ms after stimulus onset. However, relative to baseline EMG changes occurred earlier for faces (250ms) than for scenes (500ms) whereas for scenes activity changes were more pronounced over the whole viewing period. Taking into account all effects, the data suggests that emotional facial expressions evoke faster attentional orienting, but weaker affective neural activity and emotional behavioural responses compared to emotional scenes.

ORB-SLAM: A Versatile and Accurate Monocular SLAM System

This paper presents ORB-SLAM, a feature-based monocular SLAM system that operates in real time, in small and large, indoor and outdoor environments. The system is robust to severe motion clutter, allows wide baseline loop closing and relocalization, and includes full automatic initialization. Building on excellent algorithms of recent years, we designed from scratch a novel system that uses the same features for all SLAM tasks: tracking, mapping, relocalization, and loop closing. A survival of the fittest strategy that selects the points and keyframes of the reconstruction leads to excellent robustness and generates a compact and trackable map that only grows if the scene content changes, allowing lifelong operation. We present an exhaustive evaluation in 27 sequences from the most popular datasets. ORB-SLAM achieves unprecedented performance with respect to other state-of-the-art monocular SLAM approaches. For the benefit of the community, we make the source code public.

Optimal layered representation for adaptive interactive multiview video streaming

We consider an interactive multiview video streaming (IMVS) system where clients select their preferred viewpoint in a given navigation window. To provide high quality IMVS, many high quality views should be transmitted to the clients. However, this is not always possible due to the limited and heterogeneous capabilities of the clients. In this paper, we propose a novel adaptive IMVS solution based on a layered multiview representation where camera views are organized into layered subsets to match the different clients constraints. We formulate an optimization problem for the joint selection of the views subsets and their encoding rates. Then, we propose an optimal and a reduced computational complexity greedy algorithms, both based on dynamic-programming. Simulation results show the good performance of our novel algorithms compared to a baseline algorithm, proving that an effective IMVS adaptive solution should consider the scene content and the client capabilities and their preferences in navigation.

Multiscale Tikhonov-Total Variation Image Restoration Using Spatially Varying Edge Coherence Exponent.

Edge preserving regularization using partial differential equation (PDE)-based methods although extensively studied and widely used for image restoration, still have limitations in adapting to local structures. We propose a spatially adaptive multiscale variable exponent-based anisotropic variational PDE method that overcomes current shortcomings, such as over smoothing and staircasing artifacts, while still retaining and enhancing edge structures across scale. Our innovative model automatically balances between Tikhonov and total variation (TV) regularization effects using scene content information by incorporating a spatially varying edge coherence exponent map constructed using the eigenvalues of the filtered structure tensor. The multiscale exponent model we develop leads to a novel restoration method that preserves edges better and provides selective denoising without generating artifacts for both additive and multiplicative noise models. Mathematical analysis of our proposed method in variable exponent space establishes the existence of a minimizer and its properties. The discretization method we use satisfies the maximum-minimum principle which guarantees that artificial edge regions are not created. Extensive experimental results using synthetic, and natural images indicate that the proposed multiscale Tikhonov-TV (MTTV) and dynamical MTTV methods perform better than many contemporary denoising algorithms in terms of several metrics, including signal-to-noise ratio improvement and structure preservation. Promising extensions to handle multiplicative noise models and multichannel imagery are also discussed.

The occipital place area represents the local elements of scenes

Abstract Neuroimaging studies have identified three scene-selective regions in human cortex: parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA). However, precisely what scene information each region represents is not clear, especially for the least studied, more posterior OPA. Here we hypothesized that OPA represents local elements of scenes within two independent, yet complementary scene descriptors: spatial boundary (i.e., the layout of external surfaces) and scene content (e.g., internal objects). If OPA processes the local elements of spatial boundary information, then it should respond to these local elements (e.g., walls) themselves, regardless of their spatial arrangement. Indeed, we found that OPA, but not PPA or RSC, responded similarly to images of intact rooms and these same rooms in which the surfaces were fractured and rearranged, disrupting the spatial boundary. Next, if OPA represents the local elements of scene content information, then it should respond more when more such local elements (e.g., furniture) are present. Indeed, we found that OPA, but not PPA or RSC, responded more to multiple than single pieces of furniture. Taken together, these findings reveal that OPA analyzes local scene elements – both in spatial boundary and scene content representation – while PPA and RSC represent global scene properties.

Sensitivity to emotional scene content outside the focus of attention

We investigated whether the emotional content of visual scenes depicting people is processed in peripheral vision. Emotional or neutral scene photographs were paired with a matched scrambled image for 150ms in peripheral vision (≥5°). The pictures were immediately followed by a digit or letter in a discrimination task. Interference (i.e., slowed reaction times) with performance in this task indexed the processing resources drawn by the pictures. Twelve types of specific emotional scene contents (e.g., erotica or mutilation) were compared. Results showed, first, that emotional scenes caused greater interference than neutral scenes, in the absence of fixations. This suggests that emotional scenes are processed and draw covert attention outside the focus of overt attention. Second, interference was similar for female and male participants with pleasant scenes (except for erotica), but females were more affected by all types of unpleasant scenes than males. This reveals that sensitivity to peripheral vision is modulated by sex and affective valence. Third, low-level image properties, visual saliency, and size of bodies and faces, were generally equivalent for emotional and neutral scenes. This rules out the alternative hypothesis of a contribution of non-emotional, purely perceptual factors.

Iconic memory for natural scenes: Evidence using a modified change-detection procedure

ABSTRACTChange blindness for the contents of natural scenes suggests that only items that are attended while the scene is still visible are stored, leading some to characterize our visual experiences as sparse. Experiments on iconic memory for arrays of discrete symbols or objects, however, indicate observers have access to more visual information for at least several hundred milliseconds at offset of a display. In the experiment presented here, we demonstrate an iconic memory for complex natural or real-world scenes. Using a modified change detection task in which to-be changed objects are cued at offset of the scene, we show that more information from a natural scene is briefly stored than change blindness predicts and more than is contained in visual short-term memory. In our experiment, a cue appearing 0, 300, or 1000 msec after offset of the pre-change scene or at onset of the second scene presentation (a Post Cue) directed attention to the location of a possible change. Compared to a no-cue condition, subjects were significantly better at detecting changes and identifying what changed in the cue condition, with the cue having a diminishing effect as a function of time and no effect when its onset coincided with that of the second scene presentation. The results suggest that an iconic memory of a natural scene exists for at least 1000 msec after scene offset, from which subjects can access the identity of items in the pre-change scene. This implies that change blindness underestimates the amount of information available to the visual system from a brief glance of a natural scene.

Content, not context, facilitates memory for real-world scenes

ABSTRACTCan eye movements tell us whether people will remember a scene? In order to investigate the link between eye movements and memory encoding and retrieval, we asked participants to study photographs of real-world scenes while their eye movements were being tracked. We found eye gaze patterns during study to be predictive of subsequent memory for scenes. Moreover, gaze patterns during study were more similar to gaze patterns during test for remembered than for forgotten scenes. Thus, eye movements are indeed indicative of scene memory. In an explicit test for context effects of eye movements on memory, we found recognition rate to be unaffected by the disruption of spatial and/or temporal context of repeated eye movements. Therefore, we conclude that eye movements cue memory by selecting and accessing the most relevant scene content, regardless of its spatial location within the scene or the order in which it was selected.

An Ensemble Approach to Image Matching Using Contextual Features.

We propose a contextual framework for 2D image matching and registration using an ensemble feature. Our system is beneficial for registering image pairs that have captured the same scene but have large visual discrepancies between them. It is common to encounter challenging visual variations in image sets with artistic rendering differences or in those collected over a period of time during which the lighting conditions and scene content may have changed. Differences between images may also be caused using a variety of cameras with different sensors, focal lengths, and exposure values. Local feature matching techniques cannot always handle these difficulties, so we have developed an approach that builds on traditional methods to consider linear and histogram of gradient information over a larger, more stable region. We also present a technique for using linear features to estimate corner keypoints, or pseudo corners, that can be used for matching. Our pipeline follows this unique matching stage with homography refinement methods using edge and gradient information. Our goal is to increase the size of accurate keypoint match sets and align photographs containing a combination of man-made and natural imagery. We show that incorporating contextual information can provide complimentary information for scale invariant feature transform and boost local keypoint matching performance, as well as be used to describe corner feature points.

Efficient 3-D Scene Prefetching From Learning User Access Patterns

Rendering large-scale 3-D scenes on a thin client is attracting increasing attention with the development of the mobile Internet. Efficient scene prefetching to provide timely data with a limited cache is one of the most critical issues for remote 3-D data scheduling in networked virtual environment applications. Existing prefetching schemes predict the future positions of each individual user based on user traces. In this paper, we investigate scene content sequences accessed by various users instead of user viewpoint traces and propose a user access pattern-based 3-D scene prefetching scheme. We make a relationship graph-based clustering to partition history user access sequences into several clusters and choose representative sequences from among these clusters as user access patterns. Then, these user access patterns are prioritized by their popularity and users’ personal preference. Based on these access patterns, the proposed prefetching scheme predicts the scene contents that will most likely be visited in the future and delivers them to the client in advance. The experiment results demonstrate that our user access pattern-based prefetching approach achieves a high hit ratio and outperforms the prevailing prefetching schemes in terms of access latency and cache capacity.

Visual information shapes the dynamics of corticobasal ganglia pathways during response selection and inhibition.

Action selection often requires the transformation of visual information into motor plans. Preventing premature responses may entail the suppression of visual input and/or of prepared muscle activity. This study examined how the quality of visual information affects frontobasal ganglia (BG) routes associated with response selection and inhibition. Human fMRI data were collected from a stop task with visually degraded or intact face stimuli. During go trials, degraded spatial frequency information reduced the speed of information accumulation and response cautiousness. Effective connectivity analysis of the fMRI data showed action selection to emerge through the classic direct and indirect BG pathways, with inputs deriving form both prefrontal and visual regions. When stimuli were degraded, visual and prefrontal regions processing the stimulus information increased connectivity strengths toward BG, whereas regions evaluating visual scene content or response strategies reduced connectivity toward BG. Response inhibition during stop trials recruited the indirect and hyperdirect BG pathways, with input from visual and prefrontal regions. Importantly, when stimuli were nondegraded and processed fast, the optimal stop model contained additional connections from prefrontal to visual cortex. Individual differences analysis revealed that stronger prefrontal-to-visual connectivity covaried with faster inhibition times. Therefore, prefrontal-to-visual cortex connections appear to suppress the fast flow of visual input for the go task, such that the inhibition process can finish before the selection process. These results indicate response selection and inhibition within the BG to emerge through the interplay of top-down adjustments from prefrontal and bottom-up input from sensory cortex.

Motion responses in scene-selective regions

The vast majority of studies on scene processing were conducted using stationary scenes. However, during natural vision, scene views change dynamically due to self-induced eye-, head-, and body-motion, and these dynamic changes are crucial for other higher-level functions such as navigation, self-motion perception, and spatial updating. Yet, we do not know whether or how scene selective regions are modulated by visual motion and to which degree their motion response depends on scene content. In this study, we used fMRI to examine both questions using a 2×2 factorial design with the factors 2D planar motion (motion versus static) and scene content (natural scenes versus their Fourier scrambles). We found that among independently localized scene-responsive regions, parahippocampal place area (PPA) and transverse occipital sulcus (TOS), also referred to as occipital place area (OPA), were significantly motion responsive, whereas retrosplenial cortex (RSC) was not. Additionally, PPA showed an interaction between motion and scene in that it responded more to motion in context of scenes than scramble, with similar trends in TOS and RSC. These results provide a novel functional dissociation between motion-responsive PPA and TOS/OPA versus motion-unresponsive RSC and suggest a strong role for PPA in integrating motion and scene content.

Subtle differences: Sonic content in ‘translation’ remakes

Abstract This article focuses narrowly on sonic content in matching scenes from various non-English films and their adaptations, made relatively soon after their success in regional markets, by Hollywood. It makes the case that whereas the most memorable scenes in the ‘translation’ remakes typically look very much like their models, often they do not sound the same. Based on close comparisons of matching iconic scenes in Nikita and The Assassin, Infernal Affairs and The Departed, Ju-On and The Grudge, and Låt den rätte komma in and Let Me In, the article suggests that even when differences in sonic content are subtle they might still be substantial, so much so that they perhaps influence how the films in their entireties are read.

Advance Feature Extraction Technique for Digital Camera Identification

On the World Wide Web, a large amount of image content is distributed, owing to the rapid progress in digital camera technologies. For presenting evidence in a court and helping police investigations Digital camera identification, which is the identification of the source camera of input image is becoming increasingly important. A digital camera identification method using the image sensor’s pattern noise has received considerable attention in recent years. The existence of differences between the sensitivities of pixels Photo-response nonuniformity (PRNU) noise is mainly generated and it is useful as a fingerprint of a camera. By image processing engine, however the PRNU noise of an image is usually contaminated by random noise and scene content affected by the image processing engine, which inhibits stable Identification. The pairwise magnitude relations of image sensor noise, we proposed a novel digital camera identification method using, which are robust to noise contamination. That the proposed method can identify the source cameras of query images with high accuracy we demonstrate, by performing experiments.

A case study in identifying acceptable bitrates for human face recognition tasks

Face recognition from images or video footage requires a certain level of recorded image quality. This paper derives acceptable bitrates (relating to levels of compression and consequently quality) of footage with human faces, using an industry implementation of the standard H.264/MPEG-4 AVC and the Closed-Circuit Television (CCTV) recording systems on London buses. The London buses application is utilized as a case study for setting up a methodology and implementing suitable data analysis for face recognition from recorded footage, which has been degraded by compression. The majority of CCTV recorders on buses use a proprietary format based on the H.264/MPEG-4 AVC video coding standard, exploiting both spatial and temporal redundancy. Low bitrates are favored in the CCTV industry for saving storage and transmission bandwidth, but they compromise the image usefulness of the recorded imagery. In this context, usefulness is determined by the presence of enough facial information remaining in the compressed image to allow a specialist to recognize a person. The investigation includes four steps: (1) Development of a video dataset representative of typical CCTV bus scenarios. (2) Selection and grouping of video scenes based on local (facial) and global (entire scene) content properties. (3) Psychophysical investigations to identify the key scenes, which are most affected by compression, using an industry implementation of H.264/MPEG-4 AVC. (4) Testing of CCTV recording systems on buses with the key scenes and further psychophysical investigations. The results showed a dependency upon scene content properties. Very dark scenes and scenes with high levels of spatial–temporal busyness were the most challenging to compress, requiring higher bitrates to maintain useful information.

Comprehensible Video Thumbnails

AbstractWe present the Comprehensible Video Thumbnail; an automatically generated visual précis that summarizes salient objects and their dynamics within a video clip. Salient moving objects are detected within clips using a novel stochastic sampling technique that identifies, clusters and then tracks regions exhibiting affine motion coherence within the clip. Tracks are analyzed to determine salient instants at which motion and/or appearance changes significantly, and the resulting objects arranged in a stylized composition optimized to reduce visual clutter and enhance understanding of scene content through classification and depiction of motion type and trajectory. The result is an object‐level visual gist of the clip, obtained with full automation and depicting content and motion with greater descriptive power that prior approaches. We demonstrate these benefits through a user study in which the comprehension of our video thumbnails is compared to the state of the art over a wide variety of sports footage.

Dissociation of salience-driven and content-driven spatial attention to scene category with predictive decoding of gaze patterns.

Scene content is thought to be processed quickly and efficiently to bias subsequent visual exploration.Does scene content bias spatial attention during task-free visual exploration of natural scenes?If so, is this bias driven by patterns of physical salience or content-driven biases formed through previous encounters with similar scenes? We conducted two eye-tracking experiments to address these questions. Using a novel gaze decoding method, we show that fixation patterns predict scene category during free exploration. Additionally, we isolate salience-driven contributions using computational salience maps and content-driven contributions using gaze-restricted fixation data. We find distinct time courses for salience-driven and content-driven effects. The influence of physical salience peaked initially but quickly fell off at 600 ms past stimulus onset. The influence of content effects started at chance and steadily increased over the 2000 ms after stimulus onset. The combination of these two components significantly explains the time course of gaze allocation during free exploration.

Outdoor scene reflectance measurements using a Bragg-grating-based hyperspectral imager

The acquisition of spectral reflectance factor image data in an outdoor environment is a challenging task, mostly due to nonstatic scene content and illumination. In this work, we propose a work-flow for this task using a commercial Bragg-grating-based hyperspectral imager that can capture the visible and near-infrared part of the light spectrum. To our knowledge, we are the first who use this technology for outdoor spectral reflectance factor imaging. The work-flow involves focus position and exposure time estimation, illumination scaling, and image registration, among other procedures. Most of them generally apply to hyperspectral imaging, while some are specific to a Bragg-grating-based hyperspectral imaging device when dealing with specific challenges in outdoor environments. We have conducted some experiments to evaluate the quality of the acquired image data and discussed some limitations of the technology for spectral imaging of outdoor scenes. Fourteen urban scene spectral images acquired using the proposed approach are already publicly available to the scientific community under a Creative Commons license.