Objects In Natural Scenes Research Articles

Local contour features contribute to figure-ground segregation in monkey V4 neural populations and human perception

Figure-ground (FG) segregation is a crucial step towards the recognition of objects in natural scenes. Gestalt psychologists have emphasized the importance of contour features in perception of FG. Recent electrophysiological studies have identified a neural population in V4 that shows FG-dependent modulation (FG neurons). However, whether the contour features contribute to the modulation of the response patterns of the neural population remains unclear. In the present study, we quantified the contour features associated with Gestalt factors in local natural stimuli and examined whether salient contour-features evoked reliable perceptual and neural responses by analyzing response consistency (stability) across trials. The results showed the tendency that the more salient contour-features evoked the greater consistencies in the perceptual FG judgments and population-based neural responses in FG determination; a greater partial correlation for curvature and weaker correlations for closure and parallelism. Multiple linear regression analyses demonstrated that the perceptual consistency depended similarly on curvature and closure, and the neural consistency depended significantly on curvature but weakly on closure. We further observed a strong correlation between the consistencies in the perceptual and neural responses, i.e., stimuli that evoked more stable percepts tended to evoke more stable neural responses. These results indicate that local contour-features modulate the responses of the neural population in V4 and contribute to the perception of FG organization.

ETFE: Efficient transformer with feature enhancement for semantic segmentation

Abstract Despite the powerful capabilities of Transformers in relation to modeling, their effectiveness in capturing multi-scale object features in image-based tasks is often limited. This limitation arises from the diverse shapes and distances between objects in natural scenes, which pose challenges for Transformers to accurately capture object sizes and distances. To overcome this problem, we propose an effective Transformer which extracts multi-scale features at different resolutions. First, we leverage a multi-head attention mechanism to extract richer semantic features at various resolutions. The semantic features are then passed through the Feature Gate to aggregate multi-scale semantic features. Finally, Attentional Feature Enhancement further strengthens the semantic information of target features, enabling the modeling of dependencies among multi-scale features. Comprehensive experiments conducted on the Cityscapes and ADE20K datasets have shown that our model outperforms competing methods in both performance and accuracy.

A dynamic scale-mixture model of motion in natural scenes.

Some of the most important tasks of visual and motor systems involve estimating the motion of objects and tracking them over time. Such systems evolved to meet the behavioral needs of the organism in its natural environment, and may therefore be adapted to the statistics of motion it is likely to encounter. By tracking the movement of individual points in movies of natural scenes, we begin to identify common properties of natural motion across scenes. As expected, objects in natural scenes move in a persistent fashion, with velocity correlations lasting hundreds of milliseconds. More subtly, but crucially, we find that the observed velocity distributions are heavy-tailed and can be modeled as a Gaussian scale-mixture. Extending this model to the time domain leads to a dynamic scale-mixture model, consisting of a Gaussian process multiplied by a positive scalar quantity with its own independent dynamics. Dynamic scaling of velocity arises naturally as a consequence of changes in object distance from the observer, and may approximate the effects of changes in other parameters governing the motion in a given scene. This modeling and estimation framework has implications for the neurobiology of sensory and motor systems, which need to cope with these fluctuations in scale in order to represent motion efficiently and drive fast and accurate tracking behavior.

The Impact of Scene Context on Visual Object Recognition: Comparing Humans, Monkeys, and Computational Models.

During natural vision, we rarely see objects in isolation but rather embedded in rich and complex contexts. Understanding how the brain recognizes objects in natural scenes by integrating contextual information remains a key challenge. To elucidate neural mechanisms compatible with human visual processing, we need an animal model that behaves similarly to humans, so that inferred neural mechanisms can provide hypotheses relevant to the human brain. Here we assessed whether rhesus macaques could model human context-driven object recognition by quantifying visual object identification abilities across variations in the amount, quality, and congruency of contextual cues. Behavioral metrics revealed strikingly similar context-dependent patterns between humans and monkeys. However, neural responses in the inferior temporal (IT) cortex of monkeys that were never explicitly trained to discriminate objects in context, as well as current artificial neural network models, could only partially explain this cross-species correspondence. The shared behavioral variance unexplained by context-naive neural data or computational models highlights fundamental knowledge gaps. Our findings demonstrate an intriguing alignment of human and monkey visual object processing that defies full explanation by either brain activity in a key visual region or state-of-the-art models.

A new framework for improving semantic segmentation in aerial imagery

High spatial resolution (HSR) remote sensing imagery presents a rich tapestry of foreground-background intricacies, rendering semantic segmentation in aerial contexts a formidable and vital undertaking. At its core, this challenge revolves around two pivotal questions: 1) Mitigating Background Interference and Enhancing Foreground Clarity. 2) Accurate Segmentation in Dense Small Object Cluster. Conventional semantic segmentation methods primarily cater to the segmentation of large-scale objects in natural scenes, yet they often falter when confronted with aerial imagery’s characteristic traits such as vast background areas, diminutive foreground objects, and densely clustered targets. In response, we propose a novel semantic segmentation framework tailored to overcome these obstacles. To address the first challenge, we leverage PointFlow modules in tandem with the Foreground-Scene (F-S) module. PointFlow modules act as a barrier against extraneous background information, while the F-S module fosters a symbiotic relationship between the scene and foreground, enhancing clarity. For the second challenge, we adopt a dual-branch structure termed disentangled learning, comprising Foreground Precedence Estimation and Small Object Edge Alignment (SOEA). Our foreground saliency guided loss optimally directs the training process by prioritizing foreground examples and challenging background instances. Extensive experimentation on the iSAID and Vaihingen datasets validates the efficacy of our approach. Not only does our method surpass prevailing generic semantic segmentation techniques, but it also outperforms state-of-the-art remote sensing segmentation methods.

GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector.

In this paper, we present a novel end-to-end group collaborative learning network, termed GCoNet+, which can effectively and efficiently (250 fps) identify co-salient objects in natural scenes. The proposed GCoNet+ achieves the new state-of-the-art performance for co-salient object detection (CoSOD) through mining consensus representations based on the following two essential criteria: 1) intra-group compactness to better formulate the consistency among co-salient objects by capturing their inherent shared attributes using our novel group affinity module (GAM); 2) inter-group separability to effectively suppress the influence of noisy objects on the output by introducing our new group collaborating module (GCM) conditioning on the inconsistent consensus. To further improve the accuracy, we design a series of simple yet effective components as follows: i) a recurrent auxiliary classification module (RACM) promoting model learning at the semantic level; ii) a confidence enhancement module (CEM) assisting the model in improving the quality of the final predictions; and iii) a group-based symmetric triplet (GST) loss guiding the model to learn more discriminative features. Extensive experiments on three challenging benchmarks, i.e., CoCA, CoSOD3k, and CoSal2015, demonstrate that our GCoNet+ outperforms the existing 12 cutting-edge models. Code has been released at https://github.com/ZhengPeng7/GCoNet_plus.

Search templates for real-world objects in natural scenes

Search templates for real-world objects in natural scenes

Spatial and chromatic properties of numerosity estimation in isolation and context.

Numerosity estimation around the subitizing range is facilitated by a shape-template matching process and shape-coding mechanisms are selective to visual features such as colour and luminance contrast polarity. Objects in natural scenes are often embedded within other objects or textured surfaces. Numerosity estimation is improved when objects are grouped into small clusters of the same colour, a phenomenon termed groupitizing, which is thought to leverage on the subitizing system. Here we investigate whether numerosity mechanisms around the subitizing range are selective to colour, luminance contrast polarity and orientation, and how spatial organisation of context and target elements modulates target numerosity estimation. Stimuli consisted of a small number (3-to-6) of target elements presented either in isolation or embedded within context elements. To examine selectivity to colour, luminance polarity and orientation, we compared target-only conditions in which all elements were either the same or different along one of these feature dimensions. We found comparable performance in the same and different feature conditions, revealing that subitizing mechanism do not depend on ‘on-off’ luminance-polarity, colour or orientation channel interactions. We also measured the effect of varying spatial organisation of (i) context, by arranging the elements either in a grid, mirror-symmetric, translation-symmetric or random; (ii) target, by placing the elements either mirror-symmetric, on the vertices of simple shapes or random. Our results indicate higher accuracy and lower RTs in the grid compared to all other context types, with mirror symmetric, translation and random arrangements having comparable effects on target numerosity. We also found improved performance with shape-target followed by symmetric and random target arrangements in the absence and presence of context. These findings indicate that numerosity mechanisms around the subitizing range are not selective to colour, luminance polarity and orientation, and that symmetric, translation and random contexts organisations inhibit target-numerosity encoding stronger than regular/grid context.

Visual Servoing of Unknown Objects for Family Service Robots

Aiming at satisfying the increasing demand of family service robots for housework, this paper proposes a robot visual servoing scheme based on the randomized trees to complete the visual servoing task of unknown objects in natural scenes. Here, “unknown” means that there is no prior information on object models, such as template or database of the object. Firstly, an object to be manipulated is randomly selected by user prior to the visual servoing task execution. Then, the raw image information about the object can be obtained and used to train a randomized tree classifier online. Secondly, the current image features can be computed using the well-trained classifier. Finally, the visual controller can be designed according to the error of image feature, which is defined as the difference between the desired image features and current image features. Five visual positioning of unknown objects experiments, including 2D rigid object and 3D non-rigid object, are conducted on a MOTOMAN-SV3X six degree-of-freedom (DOF) manipulator robot. Experimental results show that the proposed scheme can effectively position an unknown object in complex natural scenes, such as occlusion and illumination changes. Furthermore, the developed robot visual servoing scheme has an excellent positioning accuracy within 0.05 mm positioning error.

A method to integrate and classify normal distributions.

Univariate and multivariate normal probability distributions are widely used when modeling decisions under uncertainty. Computing the performance of such models requires integrating these distributions over specific domains, which can vary widely across models. Besides some special cases where these integrals are easy to calculate, there exist no general analytical expressions, standard numerical methods, or software for these integrals. Here we present mathematical results and open-source software that provide (a) the probability in any domain of a normal in any dimensions with any parameters; (b) the probability density, cumulative distribution, and inverse cumulative distribution of any function of a normal vector; (c) the classification errors among any number of normal distributions, the Bayes-optimal discriminability index, and relation to the receiver operating characteristic (ROC); (d) dimension reduction and visualizations for such problems; and (e) tests for how reliably these methods may be used on given data. We demonstrate these tools with vision research applications of detecting occluding objects in natural scenes and detecting camouflage.

The importance of peripheral vision when searching 3D real-world scenes: A gaze-contingent study in virtual reality.

Visual search in natural scenes is a complex task relying on peripheral vision to detect potential targets and central vision to verify them. The segregation of the visual fields has been particularly established by on-screen experiments. We conducted a gaze-contingent experiment in virtual reality in order to test how the perceived roles of central and peripheral visions translated to more natural settings. The use of everyday scenes in virtual reality allowed us to study visual attention by implementing a fairly ecological protocol that cannot be implemented in the real world. Central or peripheral vision was masked during visual search, with target objects selected according to scene semantic rules. Analyzing the resulting search behavior, we found that target objects that were not spatially constrained to a probable location within the scene impacted search measures negatively. Our results diverge from on-screen studies in that search performances were only slightly affected by central vision loss. In particular, a central mask did not impact verification times when the target was grammatically constrained to an anchor object. Our findings demonstrates that the role of central vision (up to 6 degrees of eccentricities) in identifying objects in natural scenes seems to be minor, while the role of peripheral preprocessing of targets in immersive real-world searches may have been underestimated by on-screen experiments.

JU-VNT: a multi-spectral dataset of indoor object recognition using visible, near-infrared and thermal spectrum

Detecting objects in natural scenes can be a very challenging task. In several real-life scenarios it is often found that visible spectrum is not ideal for typical computer vision tasks. Going beyond the range of visible light spectrum, such as the near infrared spectrum or the thermal spectrum allows us to capture many unique properties of objects that normally not captured with a normal camera. In this work we propose two multi-spectral dataset with three different spectrum, namely, the visible, near infrared and thermal spectrum. The first dataset is a single object dataset where we have common desk objects of 25 different categories comprising of various materials. The second dataset comprises of all possible combination using these 25 objects taking a pair at a time. The objects are captured from 8 different angles using the three different cameras. The images are registered and cropped and provided along with classification and localization ground truths. Additionally classification benchmarks have been provided using the ResNet, InceptionNet and DenseNet architectures on both the datasets. The dataset would be publicly available from https://github.com/DVLP-CMATERJU/JU-VNT .

Detecting dense text in natural images

Most existing text detection methods are mainly motivated by deep learning‐based object detection approaches, which may result in serious overlapping between detected text lines, especially in dense text scenarios. It is because text boxes are not commonly overlapped, as different from general objects in natural scenes. Moreover, text detection requires higher localisation accuracy than object detection. To tackle these problems, the authors propose a novel dense text detection network (DTDN) to localise tighter text lines without overlapping. Their main novelties are: (i) propose an intersection‐over‐union overlap loss, which considers correlations between one anchor and GT boxes and measures how many text areas one anchor contains, (ii) propose a novel anchor sample selection strategy, named CMax‐OMin, to select tighter positive samples for training. CMax‐OMin strategy not only considers whether an anchor has the largest overlap with its corresponding GT box (CMax), but also ensures the overlapping between one anchor and other GT boxes as little as possible (OMin). Besides, they train a bounding‐box regressor as post‐processing to further improve text localisation performance. Experiments on scene text benchmark datasets and their proposed dense text dataset demonstrate that the proposed DTDN achieves competitive performance, especially for dense text scenarios.

On the relative (un)importance of foveal vision during letter search in naturalistic scenes

The importance of high-acuity foveal vision to visual search can be assessed by denying foveal vision using the gaze-contingent Moving Mask technique. Foveal vision was necessary to attain normal performance when searching for a target letter in alphanumeric displays, Perception & Psychophysics, 62 (2000) 576–585. In contrast, foveal vision was not necessary to correctly locate and identify medium-sized target objects in natural scenes, Journal of Experimental Psychology: Human Perception and Performance, 40 (2014) 342–360. To explore these task differences, we used grayscale pictures of real-world scenes which included a target letter (Experiment 1: T, Experiment 2: T or L). To reduce between-scene variability with regard to target salience, we developed the Target Embedding Algorithm (T.E.A.) to place the letter in a location for which there was a median change in local contrast when inserting the letter into the scene. The presence or absence of foveal vision was crossed with four target sizes. In both experiments, search performance decreased for smaller targets, and was impaired when searching the scene without foveal vision. For correct trials, the process of target localization remained completely unimpaired by the foveal scotoma, but it took longer to accept the target. We reasoned that the size of the target may affect the importance of foveal vision to the task, but the present data remain ambiguous. In summary, the data highlight the importance of extrafoveal vision for target localization, and the importance of foveal vision for target verification during letter-in-scene search.

Spectrum Characteristics Preserved Visible and Near-Infrared Image Fusion Algorithm

The visible and near-infrared images fusion aims at utilizing their spectrum characteristics to enhance visibility. However, the current visible and near-infrared fusion algorithms cannot well preserve spectrum characteristics, which results in color distortion and halo artifacts. Therefore, this paper proposes a new visible and near infrared images fusion algorithm by fully considering their different reflection and scattering characteristics. According to image degradation model, the reflection weight model and the transmission weight model are established, respectively. The reflection weight model is established by calculating the difference between the visible (red, green, and blue) spectra and the near-infrared spectrum, while maintaining the correlation of the visible spectra. The proposed reflection weight model can preserve the original reflection characteristic of objects in natural scenes. On the other hand, the transmission weight model is explicitly proposed by calculating the gradient ratio of the visible spectra to the near-infrared spectrum. The proposed transmission weight model intends to make full use of the strong transmission performance of the near-infrared spectrum, which can complement the details loss of the visible spectra caused by light scattering. Moreover, the fused image based on two models is further enhanced according to the reflection characteristics of near-infrared spectrum in case of the non-uniform illumination. The experimental results demonstrate that the proposed algorithm can not only well preserve spectrum characteristics, but also avoid color distortion while maintaining the naturalness, which outperforms the state-of-the-art.

A New Multi-Scale Convolutional Model Based on Multiple Attention for Image Classification

Computer vision systems are insensitive to the scale of objects in natural scenes, so it is important to study the multi-scale representation of features. Res2Net implements hierarchical multi-scale convolution in residual blocks, but its random grouping method affects the robustness and intuitive interpretability of the network. We propose a new multi-scale convolution model based on multiple attention. It introduces the attention mechanism into the structure of a Res2-block to better guide feature expression. First, we adopt channel attention to score channels and sort them in descending order of the feature’s importance (Channels-Sort). The sorted residual blocks are grouped and intra-block hierarchically convolved to form a single attention and multi-scale block (AMS-block). Then, we implement channel attention on the residual small blocks to constitute a dual attention and multi-scale block (DAMS-block). Introducing spatial attention before sorting the channels to form multi-attention multi-scale blocks(MAMS-block). A MAMS-convolutional neural network (CNN) is a series of multiple MAMS-blocks. It enables significant information to be expressed at more levels, and can also be easily grafted into different convolutional structures. Limited by hardware conditions, we only prove the validity of the proposed ideas through convolutional networks of the same magnitude. The experimental results show that the convolution model with an attention mechanism and multi-scale features is superior in image classification.

Age effects on the neural processing of object-context associations in briefly flashed natural scenes

In daily life, fast visual recognition of surrounding objects is facilitated through context-based expectations. However the ability to rapidly and accurately recognize unexpected stimuli in a given environment is also crucial and this ability is impaired with age. The present fMRI study aimed at comparing in young and older adults the neural correlates of fast object processing. Patterns of cerebral activity were investigated in response to briefly-presented (100 ms) congruent and incongruent natural scenes. Participants were slower and less accurate when categorizing objects in incongruent relative to congruent contexts. This behavioral cost was notably more pronounced in the older group. Height and multivariate patterns of fMRI activity in context-selective regions were equivalent in both age groups, suggesting preserved processing of coarse scene features in older participants. Incongruent scenes elicited additional activity in the parahippocampal gyrus that possibly reflected simultaneous activation of rarely co-occurring neural representations. Contextual effects were observed in object-selective cortex for the young group only, and may be driven by detection of mismatch between actually perceived and previously-experienced associations. In the older group exclusively, increased bilateral prefrontal and left fusiform activity in response to incongruent scenes was observed. However this supplemental activity was not found to efficiently contribute to improve task performance in difficult visual conditions. Altogether these results suggest age-related changes in the interaction between object- and context-processing pathways, that may subserve impairment in fast identification of unexpected objects in natural scenes.

Research on Salient Object Detection using Deep Learning and Segmentation Methods

Detecting and segmenting salient objects in natural scenes, often referred to as salient object detection has attracted a lot of interest in computer vision and recently various heuristic computational models have been designed. While many models have been proposed and several applications have emerged, yet a deep understanding of achievements and issues is lacking. The aim of this review work is to study about the details of methods in salient object detection. It not only focuses on the methods to detect saliency objects, but also reviews the works related to spatio temporal video attention detection technique in video sequences. It also discusses the open issues in terms of evaluation metrics and dataset bias in model performance and suggests future research directions. The evaluation metrics are classified into mean absolute error (MAE), Accuracy and Run-Time complexity

Association between Social Anxiety and Visual Mental Imagery of Neutral Scenes: The Moderating Role of Effortful Control.

According to cognitive theories, verbal processing attenuates emotional processing, whereas visual imagery enhances emotional processing and contributes to the maintenance of social anxiety. Individuals with social anxiety report negative mental images in social situations. However, the general ability of visual mental imagery of neutral scenes in individuals with social anxiety is still unclear. The present study investigated the general ability of non-emotional mental imagery (vividness, preferences for imagery vs. verbal processing, and object or spatial imagery) and the moderating role of effortful control in attenuating social anxiety. The participants (N = 231) completed five questionnaires. The results showed that social anxiety was not necessarily associated with all aspects of mental imagery. As suggested by theories, social anxiety was not associated with a preference for verbal processing. However, social anxiety was positively correlated with the visual imagery scale, especially the object imagery scale, which concerns the ability to construct pictorial images of individual objects. Further, it was negatively correlated with the spatial imagery scale, which concerns the ability to process information about spatial relations between objects. Although object imagery and spatial imagery positively and negatively predicted the degree of social anxiety, respectively, these effects were attenuated when socially anxious individuals had high effortful control. Specifically, in individuals with high effortful control, both object and spatial imagery were not associated with social anxiety. Socially anxious individuals might prefer to construct pictorial images of individual objects in natural scenes through object imagery. However, even in individuals who exhibit these features of mental imagery, effortful control could inhibit the increase in social anxiety.

Transformation from independent to integrative coding of multi-object arrangements in human visual cortex

To optimize processing, the human visual system utilizes regularities present in naturalistic visual input. One of these regularities is the relative position of objects in a scene (e.g., a sofa in front of a television), with behavioral research showing that regularly positioned objects are easier to perceive and to remember. Here we use fMRI to test how positional regularities are encoded in the visual system. Participants viewed pairs of objects that formed minimalistic two-object scenes (e.g., a “living room” consisting of a sofa and television) presented in their regularly experienced spatial arrangement or in an irregular arrangement (with interchanged positions). Additionally, single objects were presented centrally and in isolation. Multi-voxel activity patterns evoked by the object pairs were modeled as the average of the response patterns evoked by the two single objects forming the pair. In two experiments, this approximation in object-selective cortex was significantly less accurate for the regularly than the irregularly positioned pairs, indicating integration of individual object representations. More detailed analysis revealed a transition from independent to integrative coding along the posterior-anterior axis of the visual cortex, with the independent component (but not the integrative component) being almost perfectly predicted by object selectivity across the visual hierarchy. These results reveal a transitional stage between individual object and multi-object coding in visual cortex, providing a possible neural correlate of efficient processing of regularly positioned objects in natural scenes.