Scene Content Research Articles

Gaze position lagging behind scene content in multiple object tracking: Evidence from forward and backward presentations.

In everyday life, people often need to track moving objects. Recently, a topic of discussion has been whether people rely solely on the locations of tracked objects, or take their directions into account in multiple object tracking (MOT). In the current paper, we pose a related question: do people utilise extrapolation in their gaze behaviour, or, in more practical terms, should the mathematical models of gaze behaviour in an MOT task be based on objects' current, past or anticipated positions? We used a data-driven approach with no a priori assumption about the underlying gaze model. We repeatedly presented the same MOT trials forward and backward and collected gaze data. After reversing the data from the backward trials, we gradually tested different time adjustments to find the local maximum of similarity. In a series of four experiments, we showed that the gaze position lagged by approximately 110ms behind the scene content. We observed the lag in all subjects (Experiment 1). We further experimented to determine whether tracking workload or predictability of movements affect the size of the lag. Low workload led only to a small non-significant shortening of the lag (Experiment 2). Impairing the predictability of objects' trajectories increased the lag (Experiments 3a and 3b). We tested our observations with predictions of a centroid model: we observed a better fit for a model based on the locations of objects 110ms earlier. We conclude that mathematical models of gaze behaviour in MOT should account for the lags.

DO THE VISUAL COMPLEXITY ALGORITHMS MATCH THE GENERALIZATION PROCESS IN GEOGRAPHICAL DISPLAYS?

Abstract. In this study, we first develop a hypothesis that existing quantitative visual complexity measures will overall reflect the level of cartographic generalization, and test this hypothesis. Specifically, to test our hypothesis, we first selected common geovisualization types (i.e., cartographic maps, hybrid maps, satellite images and shaded relief maps) and retrieved examples as provided by Google Maps, OpenStreetMap and SchweizMobil by swisstopo. Selected geovisualizations vary in cartographic design choices, scene contents and different levels of generalization. Following this, we applied one of Rosenholtz et al.’s (2007) visual clutter algorithms to obtain quantitative visual complexity scores for screenshots of the selected maps. We hypothesized that visual complexity should be constant across generalization levels, however, the algorithm suggested that the complexity of small-scale displays (less detailed) is higher than those of large-scale (high detail). We also observed vast differences in visual complexity among maps providers, which we attribute to their varying approaches towards the cartographic design and generalization process. Our efforts will contribute towards creating recommendations as to how the visual complexity algorithms could be optimized for cartographic products, and eventually be utilized as a part of the cartographic design process to assess the visual complexity.

DO THE VISUAL COMPLEXITY ALGORITHMS MATCH THE GENERALIZATION PROCESS IN GEOGRAPHICAL DISPLAYS?

In this study, we first develop a hypothesis that existing quantitative visual complexity measures will overall reflect the level of cartographic generalization, and test this hypothesis. Specifically, to test our hypothesis, we first selected common geovisualization types (i.e., cartographic maps, hybrid maps, satellite images and shaded relief maps) and retrieved examples as provided by Google Maps, OpenStreetMap and SchweizMobil by swisstopo. Selected geovisualizations vary in cartographic design choices, scene contents and different levels of generalization. Following this, we applied one of Rosenholtz et al.’s (2007) visual clutter algorithms to obtain quantitative visual complexity scores for screenshots of the selected maps. We hypothesized that visual complexity should be constant across generalization levels, however, the algorithm suggested that the complexity of small-scale displays (less detailed) is higher than those of large-scale (high detail). We also observed vast differences in visual complexity among maps providers, which we attribute to their varying approaches towards the cartographic design and generalization process. Our efforts will contribute towards creating recommendations as to how the visual complexity algorithms could be optimized for cartographic products, and eventually be utilized as a part of the cartographic design process to assess the visual complexity.

SUPERPIXEL CUT FOR FIGURE-GROUND IMAGE SEGMENTATION

Figure-ground image segmentation has been a challenging problem in computer vision. Apart from the difficulties in establishing an effective framework to divide the image pixels into meaningful groups, the notions of figure and ground often need to be properly defined by providing either user inputs or object models. In this paper, we propose a novel graph-based segmentation framework, called superpixel cut. The key idea is to formulate foreground segmentation as finding a subset of superpixels that partitions a graph over superpixels. The problem is formulated as Min-Cut. Therefore, we propose a novel cost function that simultaneously minimizes the inter-class similarity while maximizing the intra-class similarity. This cost function is optimized using parametric programming. After a small learning step, our approach is fully automatic and fully bottom-up, which requires no high-level knowledge such as shape priors and scene content. It recovers coherent components of images, providing a set of multiscale hypotheses for high-level reasoning. We evaluate our proposed framework by comparing it to other generic figure-ground segmentation approaches. Our method achieves improved performance on state-of-the-art benchmark databases.

SUPERPIXEL CUT FOR FIGURE-GROUND IMAGE SEGMENTATION

Abstract. Figure-ground image segmentation has been a challenging problem in computer vision. Apart from the difficulties in establishing an effective framework to divide the image pixels into meaningful groups, the notions of figure and ground often need to be properly defined by providing either user inputs or object models. In this paper, we propose a novel graph-based segmentation framework, called superpixel cut. The key idea is to formulate foreground segmentation as finding a subset of superpixels that partitions a graph over superpixels. The problem is formulated as Min-Cut. Therefore, we propose a novel cost function that simultaneously minimizes the inter-class similarity while maximizing the intra-class similarity. This cost function is optimized using parametric programming. After a small learning step, our approach is fully automatic and fully bottom-up, which requires no high-level knowledge such as shape priors and scene content. It recovers coherent components of images, providing a set of multiscale hypotheses for high-level reasoning. We evaluate our proposed framework by comparing it to other generic figure-ground segmentation approaches. Our method achieves improved performance on state-of-the-art benchmark databases.

AUTOMATIC EXTRACTION OF BUILDING ROOF PLANES FROM AIRBORNE LIDAR DATA APPLYING AN EXTENDED 3D RANDOMIZED HOUGH TRANSFORM

This study aims to extract automatically building roof planes from airborne LIDAR data applying an extended 3D Randomized Hough Transform (RHT). The proposed methodology consists of three main steps, namely detection of building points, plane detection and refinement. For the detection of the building points, the vegetative areas are first segmented from the scene content and the bare earth is extracted afterwards. The automatic plane detection of each building is performed applying extensions of the RHT associated with additional constraint criteria during the random selection of the 3 points aiming at the optimum adaptation to the building rooftops as well as using a simple design of the accumulator that efficiently detects the prominent planes. The refinement of the plane detection is conducted based on the relationship between neighbouring planes, the locality of the point and the use of additional information. An indicative experimental comparison to verify the advantages of the extended RHT compared to the 3D Standard Hough Transform (SHT) is implemented as well as the sensitivity of the proposed extensions and accumulator design is examined in the view of quality and computational time compared to the default RHT. Further, a comparison between the extended RHT and the RANSAC is carried out. The plane detection results illustrate the potential of the proposed extended RHT in terms of robustness and efficiency for several applications.

A FRAMEWORK FOR AN AUTOMATIC SEAMLINE ENGINE

Seamline generation is a crucial last step in the ortho-image mosaicking process. In particular, it is required to convolute residual geometric and radiometric imperfections that stem from various sources. In particular, temporal differences in the acquired data will cause the scene content and illumination conditions to vary. These variations can be modelled successfully. However, one is left with micro-differences that do need to be considered in seamline generation. Another cause of discrepancies originates from the rectification surface as it will not model the actual terrain and especially human-made objects perfectly. Quality of the image orientation will also contribute to the overall differences between adjacent ortho-rectified images. <br><br> Our approach takes into consideration the aforementioned differences in designing a seamline engine. We have identified the following essential behaviours of the seamline in our engine: 1) Seamlines must pass through the path of least resistance, i.e., overlap areas with low radiometric differences. 2) Seamlines must not intersect with breaklines as that will lead to visible geometric artefacts. And finally, 3), shorter seamlines are generally favourable; they also result in faster operator review and, where necessary, interactive editing cycles. The engine design also permits alteration of the above rules for special cases. <br><br> Although our preliminary experiments are geared towards line imaging systems (i.e., the Leica ADS family), our seamline engine remains sensor agnostic. Hence, our design is capable of mosaicking images from various sources with minimal effort. The main idea behind this engine is using graph cuts which, in spirit, is based of the max-flow min-cut theory. The main advantage of using graph cuts theory is that the generated solution is global in the energy minimization sense. In addition, graph cuts allows for a highly scalable design where a set of rules contribute towards a cost function which, in turn, influences the path of minimum resistance for the seamlines. In this paper, the authors present an approach for achieving quality seamlines relatively quickly and with emphasis on generating truly seamless ortho-mosaics.

AUTOMATIC EXTRACTION OF BUILDING ROOF PLANES FROM AIRBORNE LIDAR DATA APPLYING AN EXTENDED 3D RANDOMIZED HOUGH TRANSFORM

Abstract. This study aims to extract automatically building roof planes from airborne LIDAR data applying an extended 3D Randomized Hough Transform (RHT). The proposed methodology consists of three main steps, namely detection of building points, plane detection and refinement. For the detection of the building points, the vegetative areas are first segmented from the scene content and the bare earth is extracted afterwards. The automatic plane detection of each building is performed applying extensions of the RHT associated with additional constraint criteria during the random selection of the 3 points aiming at the optimum adaptation to the building rooftops as well as using a simple design of the accumulator that efficiently detects the prominent planes. The refinement of the plane detection is conducted based on the relationship between neighbouring planes, the locality of the point and the use of additional information. An indicative experimental comparison to verify the advantages of the extended RHT compared to the 3D Standard Hough Transform (SHT) is implemented as well as the sensitivity of the proposed extensions and accumulator design is examined in the view of quality and computational time compared to the default RHT. Further, a comparison between the extended RHT and the RANSAC is carried out. The plane detection results illustrate the potential of the proposed extended RHT in terms of robustness and efficiency for several applications.

A FRAMEWORK FOR AN AUTOMATIC SEAMLINE ENGINE

Abstract. Seamline generation is a crucial last step in the ortho-image mosaicking process. In particular, it is required to convolute residual geometric and radiometric imperfections that stem from various sources. In particular, temporal differences in the acquired data will cause the scene content and illumination conditions to vary. These variations can be modelled successfully. However, one is left with micro-differences that do need to be considered in seamline generation. Another cause of discrepancies originates from the rectification surface as it will not model the actual terrain and especially human-made objects perfectly. Quality of the image orientation will also contribute to the overall differences between adjacent ortho-rectified images. Our approach takes into consideration the aforementioned differences in designing a seamline engine. We have identified the following essential behaviours of the seamline in our engine: 1) Seamlines must pass through the path of least resistance, i.e., overlap areas with low radiometric differences. 2) Seamlines must not intersect with breaklines as that will lead to visible geometric artefacts. And finally, 3), shorter seamlines are generally favourable; they also result in faster operator review and, where necessary, interactive editing cycles. The engine design also permits alteration of the above rules for special cases. Although our preliminary experiments are geared towards line imaging systems (i.e., the Leica ADS family), our seamline engine remains sensor agnostic. Hence, our design is capable of mosaicking images from various sources with minimal effort. The main idea behind this engine is using graph cuts which, in spirit, is based of the max-flow min-cut theory. The main advantage of using graph cuts theory is that the generated solution is global in the energy minimization sense. In addition, graph cuts allows for a highly scalable design where a set of rules contribute towards a cost function which, in turn, influences the path of minimum resistance for the seamlines. In this paper, the authors present an approach for achieving quality seamlines relatively quickly and with emphasis on generating truly seamless ortho-mosaics.

Single sand-dust image restoration using information loss constraint

Outdoor images captured during sand–dust weather condition typically yield poor contrast and colour shift. A novel method for single sand–dust images restoration is introduced in this paper, which relies on the atmospheric scattering model and information loss constraint. To compensate the colour shift and achieve proper luminance, the proposed atmospheric light is changing with the content of the local scenes, which is initially estimated on the basis of the general scattering model and the grey-world assumption. Then, the initial atmospheric light is updated and the coarse transmission is estimated under the information loss constraint. Next, the fast guide filter is exploited in the post-refinement process to inhibit the halo artefacts. Comparison experiments demonstrate that the proposed algorithm is straightforward and efficient, the contrast and colour shift of different kinds of sand-dust images can be well compensated, especially, nice colour fidelity and proper luminance can be maintained.

Visual high-level regions respond to high-level stimulus content in the absence of low-level confounds

High-level regions of the ventral stream exhibit strong category selectivity to stimuli such as faces, houses, or objects. However, recent studies suggest that at least part of this selectivity stems from low-level differences inherent to images of the different categories. For example, visual outdoor and indoor scenes as well as houses differ in spatial frequency, rectilinearity and obliqueness when compared to face or object images. Correspondingly, scene responsive para-hippocampal place area (PPA) showed strong preference to low-level properties of visual scenes also in the absence of high-level scene content. This raises the question whether all high-level responses in PPA, the fusiform face area (FFA), or the object-responsive lateral occipital compex (LOC) may actually be explained by systematic differences in low-level features. In the present study we contrasted two classes of simple stimuli consisting of ten rectangles each. While both were matched in visual low-level features only one class of rectangle arrangements gave rise to a percept compatible with a high-level 3D layout such as a scene or an object. We found that areas PPA, transverse occipital sulcus (TOS, also referred to as occipital place area, OPA), as well as FFA and LOC showed robust responses to the visual scene class compared to the low-level matched control. Our results suggest that visual category responsive regions are not purely driven by low-level visual features but also by the high-level perceptual stimulus interpretation.

Content Analysis using Shape and Spatial Layout with Markov Random Field

Background/Objectives: Unstructured random scene perception and understanding is a challenging problem in the field of computer vision and image processing. Methods/Statistical analysis: Two images with very different appearance may have the same color, texture and shape features. To recognize two different pictures having similar color, texture and shape we can apply spatial investigation. Findings: A methodology in the light of Markov random field has been proposed in this work to recuperate the fundamental spatial layout from a solitary image and start to examine its use as a foundation for scene understanding and content analysis in content based image retrieval. Our representation comprises of three key components (1) coarsely depicting the orientation of significant scene surfaces, (2) an examination of low level features for the understanding of image, and (3) a shape obliged Markov random field definition that enforces shape priors over the regions. Application/Improvements: We experimentally assess different Markov random field formations and exhibit the adequacy of our proposed approach in scene understanding and content analysis.

Automatic Representation and Segmentation of Video Sequences via a Novel Framework Based on the nD-EVM and Kohonen Networks

Recently in the Computer Vision field, a subject of interest, at least in almost every video application based on scene content, is video segmentation. Some of these applications are indexing, surveillance, medical imaging, event analysis, and computer-guided surgery, for naming some of them. To achieve their goals, these applications need meaningful information about a video sequence, in order to understand the events in its corresponding scene. Therefore, we need semantic information which can be obtained from objects of interest that are present in the scene. In order to recognize objects we need to compute features which aid the finding of similarities and dissimilarities, among other characteristics. For this reason, one of the most important tasks for video and image processing is segmentation. The segmentation process consists in separating data into groups that share similar features. Based on this, in this work we propose a novel framework for video representation and segmentation. The main workflow of this framework is given by the processing of an input frame sequence in order to obtain, as output, a segmented version. For video representation we use the Extreme Vertices Model in the n-Dimensional Space while we use the Discrete Compactness descriptor as feature and Kohonen Self-Organizing Maps for segmentation purposes.

Fast Quadtree Based Normalized Cross Correlation Method for Fractal Video Compression using FFT

In order to achieve fast computational speed with good visual quality of output video, we propose a frequency domain based new fractal video compression scheme. Normalized cross correlation is used to find the structural self similar domain block for the input range block. To increase the searching speed, cross correlation is implemented in the frequency domain using FFT with one computational operation for all the domain blocks instead of individual block wise calculations. The encoding time is further minimized by applying rotation and reflection DFT properties to the IFFT of zero padded range blocks. The energy of overlap small size domain blocks is pre-computed for the entire reference frame and retaining the energies of the overlapped search window portion of previous adjacent block. Quadtree decompositions are obtained by using domain block motion compensated prediction error as a threshold to control the further partitions of the block. It provides a better level of adaption to the scene contents than fixed block size approach. The result shows that, on average, the proposed method can raise the encoding speed by 48.8 % and 90 % higher than NHEXS and CPM/NCIM algorithms respectively. The compression ratio and PSNR of the proposed method is increased by 15.41 and 0.89 dB higher than that of NHEXS on average. For low bit rate videos, the proposed algorithm achieve the high compression ratio above 120 with more than 31 dB PSNR.

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Depth-image-based rendering (DIBR) is a commonly used method for synthesizing additional views using video-plus-depth (V+D) format. A critical issue with DIBR-based view synthesis is the lack of information behind foreground objects. This lack is manifested as disocclusions, holes, next to the foreground objects in rendered virtual views as a consequence of the virtual camera “seeing” behind the foreground object. The disocclusions are larger in the extrapolation case, i.e. the single camera case. Texture synthesis methods (inpainting methods) aim to fill these disocclusions by producing plausible texture content. However, virtual views inevitably exhibit both spatial and temporal inconsistencies at the filled disocclusion areas, depending on the scene content. In this paper, we propose a layered depth image (LDI) approach that improves the spatio-temporal consistency. In the process of LDI generation, depth information is used to classify the foreground and background in order to form a static scene sprite from a set of neighboring frames. Occlusions in the LDI are then identified and filled using inpainting, such that no disocclusions appear when the LDI data is rendered to a virtual view. In addition to the depth information, optical flow is computed to extract the stationary parts of the scene and to classify the occlusions in the inpainting process. Experimental results demonstrate that spatio-temporal inconsistencies are significantly reduced using the proposed method. Furthermore, subjective and objective qualities are improved compared to state-of-the-art reference methods.

The occipital place area represents the local elements of scenes

Neuroimaging studies have identified three scene-selective regions in human cortex: parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA). However, precisely what scene information each region represents is not clear, especially for the least studied, more posterior OPA. Here we hypothesized that OPA represents local elements of scenes within two independent, yet complementary scene descriptors: spatial boundary (i.e., the layout of external surfaces) and scene content (e.g., internal objects). If OPA processes the local elements of spatial boundary information, then it should respond to these local elements (e.g., walls) themselves, regardless of their spatial arrangement. Indeed, we found that OPA, but not PPA or RSC, responded similarly to images of intact rooms and these same rooms in which the surfaces were fractured and rearranged, disrupting the spatial boundary. Next, if OPA represents the local elements of scene content information, then it should respond more when more such local elements (e.g., furniture) are present. Indeed, we found that OPA, but not PPA or RSC, responded more to multiple than single pieces of furniture. Taken together, these findings reveal that OPA analyzes local scene elements – both in spatial boundary and scene content representation – while PPA and RSC represent global scene properties.

Luminance-dependent long-term chromatic adaptation.

There is theoretical and empirical support for long-term adaptation of human vision to chromatic regularities in the environment. The current study investigates whether relationships of luminance and chromaticity in the natural environment could drive chromatic adaptation independently and differently for bright and dark colors. This is motivated by psychophysical evidence of systematic difference shifts in red-green chromatic sensitivities between contextually bright- versus dark-colored stimuli. For some broad classes of scene content, consistent shifts in chromaticity are found between high and low light levels within images. Especially in those images in which sky and terrain are juxtaposed, this shift has direction and magnitude consistent with the observed psychophysical shifts in the red-green balance between bright and dark colors. Taken together, these findings suggest that relative weighting of M- and L-cone signals could be adapted, in a luminance-dependent fashion, to regularities in the natural environment.

Lateral and Medial Ventral Occipitotemporal Regions Interact During the Recognition of Images Revealed from Noise.

Several studies suggest different functional roles for the medial and the lateral sections of the ventral visual cortex in object recognition. Texture and surface information is processed in medial sections, while shape information is processed in lateral sections. This begs the question whether and how these functionally specialized sections interact with each other and with early visual cortex to facilitate object recognition. In the current research, we set out to answer this question. In an fMRI study, 13 subjects viewed and recognized images of objects and animals that were gradually revealed from noise while their brains were being scanned. We applied dynamic causal modeling (DCM)—a method to characterize network interactions—to determine the modulatory effect of object recognition on a network comprising the primary visual cortex (V1), the lingual gyrus (LG) in medial ventral cortex and the lateral occipital cortex (LO). We found that object recognition modulated the bilateral connectivity between LG and LO. Moreover, the feed-forward connectivity from V1 to LG and LO was modulated, while there was no evidence for feedback from these regions to V1 during object recognition. In particular, the interaction between medial and lateral areas supports a framework in which visual recognition of objects is achieved by networked regions that integrate information on image statistics, scene content and shape—rather than by a single categorically specialized region—within the ventral visual cortex.

Combined Haze and Cirrus Removal for Multispectral Imagery

Multispectral satellite images are often contaminated by haze and/or cirrus. A previous paper presented a haze removal method that calculates a haze thickness map (HTM) based on a local search of dark objects. The haze-free signal is restored by subtracting the HTM from the hazy image assuming an additive model of the haze influence. The HTM method is substantially improved by employing the 1.38- $\mu\mbox{m}$ cirrus band. The top-of-atmosphere reflectance cirrus band is used as an additional source of information. The method restores the information in highly inhomogeneous surfaces attenuated by a low-altitude haze and high-altitude cirrus, improving the interpretation of the scene content while preserving the shape of the spectral signatures. The new enhanced HTM method is successfully applied to Landsat-8 OLI and Sentinel-2 real and simulated scenes.

Context-Based Scene Understanding

Context plays an important role in performance of object detection. There are two popular considerations in building context models for computer vision applications; type of context (semantic, spatial, scale) and scope of the relations (pairwise, high-order). In this paper, a new unified framework is presented that combines multiple sources of context in high-order relations to encode semantical coherence and consistency of the scenes. This framework introduces a new descriptor called context relevance score to model context-based distribution of the response variables and apply it to two distributions. First model incorporates context descriptor along with annotation response into a supervised Latent Dirichlet Allocation (LDA) built on multi-variate Bernoulli distribution called Context-Based LDA (CBLDA). The second model is based on multi-variate Wallenius' non-central Hyper-geometric distribution and is called Wallenius LDA (WLDA). WLDA incorporates context knowledge as bias parameter. Scene context is modeled as a graph and effectively used in object detection framework to maximize semantical consistency of the scene. The graph can also be used in recognition of out-of-context objects. Annotation metadata of Sun397 dataset is used to construct the context model. Performance of the proposed approaches was evaluated on ImageNet dataset. Comparison between proposed approaches and state-of-art multi-class object annotation algorithm shows superiority of presented approach in labeling of scene content.