Objects In Real-world Scenes Research Articles

Anchor objects drive realism while diagnostic objects drive categorization in GAN generated scenes

Our visual surroundings are highly complex. Despite this, we understand and navigate them effortlessly. This requires transforming incoming sensory information into representations that not only span low- to high-level visual features (e.g., edges, object parts, objects), but likely also reflect co-occurrence statistics of objects in real-world scenes. Here, so-called anchor objects are defined as being highly predictive of the location and identity of frequently co-occuring (usually smaller) objects, derived from object clustering statistics in real-world scenes, while so-called diagnostic objects are predictive of the larger semantic context (i.e., scene category). Across two studies (N1 = 50, N2 = 44), we investigate which of these properties underlie scene understanding across two dimensions – realism and categorisation – using scenes generated from Generative Adversarial Networks (GANs) which naturally vary along these dimensions. We show that anchor objects and mainly high-level features extracted from a range of pre-trained deep neural networks (DNNs) drove realism both at first glance and after initial processing. Categorisation performance was mainly determined by diagnostic objects, regardless of realism, at first glance and after initial processing. Our results are testament to the visual system’s ability to pick up on reliable, category specific sources of information that are flexible towards disturbances across the visual feature-hierarchy.

RCNet: Related Context-Driven Network with Hierarchical Attention for Salient Object Detection

Recent progress in salient object detection (SOD) mainly depends on dilated convolution with different receptive fields to capture contextual information for multi-scale learning. Intuitively, contextual information in different scales is conducive to understanding the image content, and thus can help us identify and locate salient objects in real-world scenes. However, the sparsity inside the dilated convolution kernel may cause the problem of local information loss, limiting the predictive accuracy of the model. In addition, the inequality of feature channels should also be considered, and they often feature different deviations for salient objects or background noises. Although some channel attention mechanisms have been proposed in SOD, their ability to capture global information is limited, and the problem of high complexity is still a great challenge. To alleviate the abovementioned problems, we propose a Related Context-Driven Network (RCNet) with Hierarchical Attention for Salient Object Detection, consisting of a cascaded multi-scale context exploration (CMCE) module and a hierarchical feature aggregation (HFA) module. The CMCE module is to capture multi-scale contextual information through using multi-receptive-field dilated convolutions in a diamond hierarchical structure, where a feature reconstruction operation is deployed to improve the correlation of features, effectively avoiding the gridding problems and local information loss. Meanwhile, the HFA module adaptively interacts with the complementary information of the multi-level features to further capture the important information from within the feature channel by a multi-source hybrid channel attention (MHCA) mechanism to generate powerful and robust feature representations. Extensive experiments on six benchmark datasets demonstrate that the proposed RCNet method consistently outperforms 20 existing the state-of-the-art SOD methods in terms of accuracy, generalization capacity and robustness.

Aging attenuates the memory advantage for unexpected objects in real-world scenes

Across the adult lifespan memory processes are subject to pronounced changes. Prior knowledge and expectations might critically shape functional differences; however, corresponding findings have remained ambiguous so far. Here, we chose a tailored approach to scrutinize how schema (in-)congruencies affect older and younger adults’ memory for objects embedded in real-world scenes, a scenario close to everyday life memory demands. A sample of 23 older (52–81 years) and 23 younger adults (18–38 years) freely viewed 60 photographs of scenes in which target objects were included that were either congruent or incongruent with the given context information. After a delay, recognition performance for those objects was determined. In addition, recognized objects had to be matched to the scene context in which they were previously presented. While we found schema violations beneficial for object recognition across age groups, the advantage was significantly less pronounced in older adults. We moreover observed an age-related congruency bias for matching objects to their original scene context. Our findings support a critical role of predictive processes for age-related memory differences and indicate enhanced weighting of predictions with age. We suggest that recent predictive processing theories provide a particularly useful framework to elaborate on age-related functional vulnerabilities as well as stability.

Long-term memory for objects in real-world scenes: The effects of semantic consistency and task priorities

Long-term memory for objects in real-world scenes: The effects of semantic consistency and task priorities

Occlusion impairs numerical discrimination of objects in real-world scenes

Occlusion impairs numerical discrimination of objects in real-world scenes

An active foveated gaze prediction algorithm based on a Bayesian ideal observer

Predicting human eye movements is a crucial task for understanding human behavior and has numerous applications in machine vision. Most current models for predicting eye movements are data-driven and require large datasets of recorded eye movements, which can be expensive and time-consuming to collect. In this paper, we present a novel theory-based model for predicting eye movements in a foveated visual system that maximizes information gain at each fixation. Our model uses a region-proposal network and eccentricity-based max pooling to account for the loss of detail in peripheral vision. We apply our model to predict human fixations in a visual search task for objects in real-world scenes. Unlike data-driven models, our model does not require training on large eye movement datasets and can generalize to any set of natural images and targets. We evaluate the generalization capability of our model by demonstrating its results on two publicly available visual search datasets, Ehinger and COCO-search18, without any further training on those datasets. Our model outperforms or performs comparably to data-driven models that are directly trained on human eye movement datasets.

Hierarchical relations of objects in real-world scenes

Hierarchical relations of objects in real-world scenes

Meaning and expected surfaces combine to guide attention during visual search in scenes.

How do spatial constraints and meaningful scene regions interact to control overt attention during visual search for objects in real-world scenes? To answer this question, we combined novel surface maps of the likely locations of target objects with maps of the spatial distribution of scene semantic content. The surface maps captured likely target surfaces as continuous probabilities. Meaning was represented by meaning maps highlighting the distribution of semantic content in local scene regions. Attention was indexed by eye movements during the search for target objects that varied in the likelihood they would appear on specific surfaces. The interaction between surface maps and meaning maps was analyzed to test whether fixations were directed to meaningful scene regions on target-related surfaces. Overall, meaningful scene regions were more likely to be fixated if they appeared on target-related surfaces than if they appeared on target-unrelated surfaces. These findings suggest that the visual system prioritizes meaningful scene regions on target-related surfaces during visual search in scenes.

Effects of prior knowledge on memory for objects in real-world scenes: Schema violations benefit memory and metacognitive performance

Effects of prior knowledge on memory for objects in real-world scenes: Schema violations benefit memory and metacognitive performance

EEG signatures of contextual influences on visual search with real scenes.

The use of scene context is a powerful way by which biological organisms guide and facilitate visual search. Although many studies have shown enhancements of target-related electroencephalographic activity (EEG) with synthetic cues, there have been fewer studies demonstrating such enhancements during search with scene context and objects in real world scenes. Here, observers covertly searched for a target in images of real scenes while we used EEG to measure the steady state visual evoked response to objects flickering at different frequencies. The target appeared in its typical contextual location or out of context while we controlled for low-level properties of the image including target saliency against the background and retinal eccentricity. A pattern classifier using EEG activity at the relevant modulated frequencies showed target detection accuracy increased when the target was in a contextually appropriate location. A control condition for which observers searched the same images for a different target orthogonal to the contextual manipulation, resulted in no effects of scene context on classifier performance, confirming that image properties cannot explain the contextual modulations of neural activity. Pattern classifier decisions for individual images were also related to the aggregated observer behavioral decisions for individual images. Together, these findings demonstrate target-related neural responses are modulated by scene context during visual search with real world scenes and can be related to behavioral search decisions.

3D Distance Measurement from a Camera to a Mobile Vehicle, Using Monocular Vision

Estimation of distance from objects in real‐world scenes is an important topic in several applications such as navigation of autonomous robots, simultaneous localization and mapping (SLAM), and augmented reality (AR). Even though there is a technology for this purpose, in some cases, this technology has some disadvantages. For example, GPS systems are susceptible to interference, especially in places surrounded by buildings, under bridges or indoors; alternatively, RGBD sensors can be used, but they are expensive, and their operational range is limited. Monocular vision is a low‐cost suitable alternative that can be used indoor and outdoor. However, monocular odometry is challenging because the object location can be known up a scale factor. Moreover, when objects are moving, it is necessary to estimate the location from consecutive images accumulating error. This paper introduces a new method to compute the distance from a single image of the desired object, with known dimensions, captured with a monocular calibrated vision system. This method is less restrictive than other proposals in the state‐of‐the‐art literature. For the detection of interest points, a Region‐based Convolutional Neural Network combined with a corner detector were used. The proposed method was tested on a standard dataset and images acquired by a low‐cost and low‐resolution webcam, under noncontrolled conditions. The system was tested and compared with a calibrated stereo vision system. Results showed the similar performance of both systems, but the monocular system accomplished the task in less time.

Expecting and detecting objects in real-world scenes: when do target, nontarget and coarse scene features contribute?

Expecting and detecting objects in real-world scenes: when do target, nontarget and coarse scene features contribute?

Change detection of meaningful objects in real-world scenes in adolescents with and without autism spectrum disorder.

Previous research suggested that adolescents with autism spectrum disorder are better than typically developing children in detecting local, non-social details within complex visual scenes. To better understand these differences, we used the image database by Sareen et al., containing the size and on-screen location information of all changes in the images, in a change blindness paradigm. In this task, an original and a modified real-world scene, separated by a gray blank, alternate repeatedly until observers detect the change. Our results indicated that participants with and without autism spectrum disorder performed similarly when scenes were presented upright, but that only the performance of the typically developing adolescents became worse in the inverted scene condition. In this condition, the correlation between performance and both image difficulty and change predictability was significantly weaker in autism spectrum disorder than in typically developing participants. We suggest that these findings result from a more locally biased search strategy in people with autism spectrum disorder, compared to typically developing participants, in tasks in which the rapid processing of global information does not help to improve change detection performance. Finally, although we found change location, change size, and age to influence participant performance, none of these was directly linked to the observed group-level differences.

Online Multi-Target Tracking Using Recurrent Neural Networks

We present a novel approach to online multi-target tracking based on recurrent neural networks (RNNs). Tracking multiple objects in real-world scenes involves many challenges, including a) an a-priori unknown and time-varying number of targets, b) a continuous state estimation of all present targets, and c) a discrete combinatorial problem of data association. Most previous methods involve complex models that require tedious tuning of parameters. Here, we propose for the first time, an end-to-end learning approach for online multi-target tracking. Existing deep learning methods are not designed for the above challenges and cannot be trivially applied to the task. Our solution addresses all of the above points in a principled way. Experiments on both synthetic and real data show promising results obtained at ~300 Hz on a standard CPU, and pave the way towards future research in this direction.

On the visual span during object search in real-world scenes

The current study investigated from how large a region around their current point of gaze viewers can take in information when searching for objects in real-world scenes. Visual span size was estimated using the gaze-contingent moving window paradigm. Experiment 1 featured window radii measuring 1, 3, 4, 4.7, 5.4, and 6.1°. Experiment 2 featured six window radii measuring between 5 and 10°. Each scene occupied a 24.8 × 18.6° field of view. Inside the moving window, the scene was presented in high resolution. Outside the window, the scene image was low-pass filtered to impede the parsing of the scene into constituent objects. Visual span was defined as the window size at which object search times became indistinguishable from search times in the no-window control condition; this occurred with windows measuring 8° and larger. Notably, as long as central vision was fully available (window radii ≥ 5°), the distance traversed by the eyes through the scene to the search target was comparable to baseline performance. However, to move their eyes to the target, viewers made shorter saccades, requiring more fixations to cover the same image space, and thus more time. Moreover, a gaze-data based decomposition of search time revealed disruptions in specific subprocesses of search. In addition, nonlinear mixed models analyses demonstrated reliable individual differences in visual span size and parameters of the search time function.

Decisions about objects in real-world scenes are influenced by visual saliency before and during their inspection

Evidence from eye-tracking experiments has provided mixed support for saliency map models of inspection, with the task set for the viewer accounting for some of the discrepancies between predictions and observations. In the present experiment viewers inspected pictures of road scenes with the task being to decide whether or not they would enter a highway from a junction. Road safety observations have concluded that highly visible road users are less likely to be involved in crashes, suggesting that saliency is important in real-world tasks. The saliency of a critical vehicle was varied in the present task, as was the type of vehicle and the preferred vehicle of the viewer. Decisions were influenced by saliency, with more risky decisions when low saliency motorcycles were present. Given that the vehicles were invariably inspected, this may relate to the high incidence of “looked-but-failed-to-see” crashes involving motorcycles and to prevalence effects in visual search. Eye-tracking measures indicated effects of saliency on the fixation preceding inspection of the critical vehicle (as well as effects on inspection of the vehicle itself), suggesting that high saliency can attract an early fixation. These results have implications for recommendations about the conspicuity of vulnerable road users.

Global image properties do not guide visual search

While basic visual features such as color, motion, and orientation can guide attention, it is likely that additional features guide search for objects in real-world scenes. Recent work has shown that human observers efficiently extract global scene properties such as mean depth or navigability from a brief glance at a single scene (M. R. Greene & A. Oliva, 2009a, 2009b). Can human observers also efficiently search for an image possessing a particular global scene property among other images lacking that property? Observers searched for scene image targets defined by global properties of naturalness, transience, navigability, and mean depth. All produced inefficient search. Search efficiency for a property was not correlated with its classification threshold time from M. R. Greene and A. Oliva (2009b). Differences in search efficiency between properties can be partially explained by low-level visual features that are correlated with the global property. Overall, while global scene properties can be rapidly classified from a single image, it does not appear to be possible to use those properties to guide attention to one of several images.

Semantic memory for contextual regularities within and across scene categories: Evidence from eye movements

When encountering familiar scenes, observers can use item-specific memory to facilitate the guidance of attention to objects appearing in known locations or configurations. Here, we investigated how memory for relational contingencies that emerge across different scenes can be exploited to guide attention. Participants searched for letter targets embedded in pictures of bedrooms. In a between-subjects manipulation, targets were either always on a bed pillow or randomly positioned. When targets were systematically located within scenes, search for targets became more efficient. Importantly, this learning transferred to bedrooms without pillows, ruling out learning that is based on perceptual contingencies. Learning also transferred to living room scenes, but it did not transfer to kitchen scenes, even though both scene types contained pillows. These results suggest that statistical regularities abstracted across a range of stimuli are governed by semantic expectations regarding the presence of target-predicting local landmarks. Moreover, explicit awareness of these contingencies led to a central tendency bias in recall memory for precise target positions that is similar to the spatial category effects observed in landmark memory. These results broaden the scope of conditions under which contextual cuing operates and demonstrate how semantic memory plays a causal and independent role in the learning of associations between objects in real-world scenes.

Peripheral processing of objects in real-world scenes

Peripheral processing of objects in real-world scenes

Attention to smoking-related and incongruous objects during scene viewing

This study examined the influences of semantic characteristics of objects in real-world scenes on allocation of attention as reflected in eye movement measures. Stimuli consisted of full-color photographic scenes, and within each scene, the semantic salience of two target objects was manipulated while the objects’ perceptual salience was kept constant. One of the target objects was either inconsistent or consistent with the scene category. In addition, the second target object was either smoking-related or neutral. Two groups of college students, namely current cigarette smokers (N=18) and non-smokers (N=19), viewed each scene for 10s while their eye movements were recorded. While both groups showed preferential allocation of attention to inconsistent objects, smokers also selectively attended to smoking-related objects. Theoretical implications of the results are discussed.