Figure-ground Segmentation Research Articles

Figure-ground segmentation based on motion in the archerfish.

Figure-ground segmentation is a fundamental process in visual perception that involves separating visual stimuli into distinct meaningful objects and their surrounding context, thus allowing the brain to interpret and understand complex visual scenes. Mammals exhibit varying figure-ground segmentation capabilities, ranging from primates that can perform well on figure-ground segmentation tasks to rodents that perform poorly. To explore figure-ground segmentation capabilities in teleost fish, we studied how the archerfish, an expert visual hunter, performs figure-ground segmentation. We trained archerfish to discriminate foreground objects from the background, where the figures were defined by motion as well as by discontinuities in intensity and texture. Specifically, the figures were defined by grating, naturalistic texture, and random noise moving in counterphase with the background. The archerfish performed the task well and could distinguish between all three types of figures and grounds. Their performance was comparable to that of primates and outperformed rodents. These findings suggest the existence of a complex visual process in the archerfish visual system that enables the delineation of figures as distinct from backgrounds, and provide insights into object recognition in this animal.

Figure-Ground Segmentation and Biological Motion Perception in Peripheral Visual Field.

Biological motion perception is a specific type of perceptual organization, during which a clear image of a moving human body is perceptually generated in virtue of certain core light dots representing the major joint movements. While the processes of biological motion perception have been studied extensively for almost a century, there is still a debate on whether biological motion task performance can be equally precise across all visual field or is central visual field specified for biological motion perception. The current study explores the processes of biological motion perception and figure-ground segmentation in the central and peripheral visual field, expanding the understanding of perceptual organization across different eccentricities. The method involved three different tasks of visual grouping: (1) a static visual grouping task, (2) a dynamic visual grouping task, and (3) a biological motion detection task. The stimuli in (1) and (2) were generated from 12-13 dots grouped by proximity and common fate, and, in (3), light dots representing human motion. All stimuli were embedded in static or dynamics visual noise and the threshold value for the number of noise dots in which the elements could still be grouped by proximity and/or common fate was determined. The results demonstrate that biological motion can be differentiated from the scrambled set of moving dots in a more intensive visual noise than static and dynamic visual grouping tasks. Furthermore, in all three visual tasks (static and dynamic grouping, and biological motion detection) the performance was significantly worse in the periphery than in the central visual field, and object magnification could not compensate for the reduced performance in any of the three grouping tasks. The preliminary results of nine participants indicate that (a) human motion perception involves specific perceptual processes, providing the high-accuracy perception of the human body and (b) the processes of figure-ground segmentation are governed by the bottom-up processes and the best performance can be achieved only when the object is demonstrated in the central visual field.

Mice and primates use distinct strategies for visual segmentation.

The rodent visual system has attracted great interest in recent years due to its experimental tractability, but the fundamental mechanisms used by the mouse to represent the visual world remain unclear. In the primate, researchers have argued from both behavioral and neural evidence that a key step in visual representation is 'figure-ground segmentation', the delineation of figures as distinct from backgrounds. To determine if mice also show behavioral and neural signatures of figure-ground segmentation, we trained mice on a figure-ground segmentation task where figures were defined by gratings and naturalistic textures moving counterphase to the background. Unlike primates, mice were severely limited in their ability to segment figure from ground using the opponent motion cue, with segmentation behavior strongly dependent on the specific carrier pattern. Remarkably, when mice were forced to localize naturalistic patterns defined by opponent motion, they adopted a strategy of brute force memorization of texture patterns. In contrast, primates, including humans, macaques, and mouse lemurs, could readily segment figures independent of carrier pattern using the opponent motion cue. Consistent with mouse behavior, neural responses to the same stimuli recorded in mouse visual areas V1, RL, and LM also did not support texture-invariant segmentation of figures using opponent motion. Modeling revealed that the texture dependence of both the mouse's behavior and neural responses could be explained by a feedforward neural network lacking explicit segmentation capabilities. These findings reveal a fundamental limitation in the ability of mice to segment visual objects compared to primates.

A Critical Test of Deep Convolutional Neural Networks' Ability to Capture Recurrent Processing in the Brain Using Visual Masking.

Recurrent processing is a crucial feature in human visual processing supporting perceptual grouping, figure-ground segmentation, and recognition under challenging conditions. There is a clear need to incorporate recurrent processing in deep convolutional neural networks, but the computations underlying recurrent processing remain unclear. In this article, we tested a form of recurrence in deep residual networks (ResNets) to capture recurrent processing signals in the human brain. Although ResNets are feedforward networks, they approximate an excitatory additive form of recurrence. Essentially, this form of recurrence consists of repeating excitatory activations in response to a static stimulus. Here, we used ResNets of varying depths (reflecting varying levels of recurrent processing) to explain EEG activity within a visual masking paradigm. Sixty-two humans and 50 artificial agents (10 ResNet models of depths -4, 6, 10, 18, and 34) completed an object categorization task. We show that deeper networks explained more variance in brain activity compared with shallower networks. Furthermore, all ResNets captured differences in brain activity between unmasked and masked trials, with differences starting at ∼98 msec (from stimulus onset). These early differences indicated that EEG activity reflected "pure" feedforward signals only briefly (up to ∼98 msec). After ∼98 msec, deeper networks showed a significant increase in explained variance, which peaks at ∼200 msec, but only within unmasked trials, not masked trials. In summary, we provided clear evidence that excitatory additive recurrent processing in ResNets captures some of the recurrent processing in humans.

A mighty tool not only in perception: Figure-ground mechanisms control binding and retrieval alike

Stimulus and response features are linked together into an event file when a response is made towards a stimulus. If some or all linked features repeat, the whole event file (including the previous response) is retrieved, thereby affecting current performance (as measured in so-called binding effects). Applying the figure-ground segmentation principle to such action control experiments, previous research showed that only stimulus features that have a figure-like character led to binding effects, while features in the background did not. Against the background of recent theorizing, integration and retrieval are discussed as separate processes that independently contribute to binding effects (BRAC framework). Thus, previous research did not specify whether figure-ground manipulations exert their modulating influence on integration and/or retrieval. We tested this in three experiments. Participants worked through a sequential distractor-response binding (DRB) task, allowing measurement of binding effects between responses and distractor (color) features. Importantly, we manipulated whether the distractor color was presented as a background feature or as a figure feature. In contrast to previous experiments, we applied this manipulation only to prime displays (Experiment 1), only to probe display (Experiment 2), or varied the figure-ground manipulation orthogonally for primes and probes (Experiment 3). Together the results of all three experiments suggest that figure-ground segmentation affects DRB effects on top of encoding specificity, and that especially the retrieval process is affected by this manipulation.

Spiking model of fixational eye movements and figure-ground segmentation

ABSTRACT We present a model connecting eye movements and cortical state. Its structure includes simulated retinal images, motion detection, feature detectors and layers of spiking neurons. The designed scheme shows how the effect of micro-saccadic scale eye movements can lead to successful figure segregation in a figure-ground paradigm, by inducing changes in the neural dynamics through the time evolution of the inhibition range.

On the Necessity of Recurrent Processing during Object Recognition: It Depends on the Need for Scene Segmentation.

Although feedforward activity may suffice for recognizing objects in isolation, additional visual operations that aid object recognition might be needed for real-world scenes. One such additional operation is figure-ground segmentation, extracting the relevant features and locations of the target object while ignoring irrelevant features. In this study of 60 human participants (female and male), we show objects on backgrounds of increasing complexity to investigate whether recurrent computations are increasingly important for segmenting objects from more complex backgrounds. Three lines of evidence show that recurrent processing is critical for recognition of objects embedded in complex scenes. First, behavioral results indicated a greater reduction in performance after masking objects presented on more complex backgrounds, with the degree of impairment increasing with increasing background complexity. Second, electroencephalography (EEG) measurements showed clear differences in the evoked response potentials between conditions around time points beyond feedforward activity, and exploratory object decoding analyses based on the EEG signal indicated later decoding onsets for objects embedded in more complex backgrounds. Third, deep convolutional neural network performance confirmed this interpretation. Feedforward and less deep networks showed a higher degree of impairment in recognition for objects in complex backgrounds compared with recurrent and deeper networks. Together, these results support the notion that recurrent computations drive figure-ground segmentation of objects in complex scenes.SIGNIFICANCE STATEMENT The incredible speed of object recognition suggests that it relies purely on a fast feedforward buildup of perceptual activity. However, this view is contradicted by studies showing that disruption of recurrent processing leads to decreased object recognition performance. Here, we resolve this issue by showing that how object recognition is resolved and whether recurrent processing is crucial depends on the context in which it is presented. For objects presented in isolation or in simple environments, feedforward activity could be sufficient for successful object recognition. However, when the environment is more complex, additional processing seems necessary to select the elements that belong to the object and by that segregate them from the background.

How inhibitory neurons increase information transmission under threshold modulation.

SUMMARYModulation of neuronal thresholds is ubiquitous in the brain. Phenomena such as figure-ground segmentation, motion detection, stimulus anticipation, and shifts in attention all involve changes in a neuron’s threshold based on signals from larger scales than its primary inputs. However, this modulation reduces the accuracy with which neurons can represent their primary inputs, creating a mystery as to why threshold modulation is so widespread in the brain. We find that modulation is less detrimental than other forms of neuronal variability and that its negative effects can be nearly completely eliminated if modulation is applied selectively to sparsely responding neurons in a circuit by inhibitory neurons. We verify these predictions in the retina where we find that inhibitory amacrine cells selectively deliver modulation signals to sparsely responding ganglion cell types. Our findings elucidate the central role that inhibitory neurons play in maximizing information transmission under modulation.

Segmentation of White Chali Arecanuts using Soft Computing Methods

Performance of computer vision based grading systems is remarkably affected by the efficiency of object segmentation. The automatic segmentation of low contrast objects is a challenging task in various fruit and nut grading systems. In this paper background elimination of white chali arecanut images is carried out using morphological segmentation. The fine-tuning of edge threshold for morphological segmentation is achieved by obtaining threshold values from multilevel thresholding of original grayscale image. The best figure ground segmentation is selected by a network trained using shape parameters of the ground truth masks. The performance of morphological segmentation is evaluated for the best figure ground segmentations using precision, recall and F-scores. Comparison of segmentation performance is done by employing multilevel thresholding based on Otsu, Fuzzy c-mean, Harmony search, Differential Evolution and Cuckoo Search algorithms. The experimental result shows that, multilevel thresholding using Differential Evolution and Cuckoo Search algorithms yield best results for the fine-tuning of edge thresholds and hence the better segmentation performance of the white chali arecanuts.

Genetic programming based feature construction methods for foreground object segmentation

Foreground object segmentation is a crucial preprocessing step for many high-level computer vision tasks, e.g. object recognition. It is still challenging to achieve accurate segmentation, especially for complex images (e.g. with high variations). Feature construction can help to improve the segmentation performance by extracting more distinctive features for foreground/background regions from the original features. However, commonly-used feature construction methods (e.g. principle component analysis) often involve certain assumptions/constraints, and the constructed features cannot be interpreted. To address these problems, genetic programming (GP) is employed in this paper, which is a well-suited feature construction technique.The aim of this work is to design new feature construction methods using GP, and analyse/compare popular GP-based feature construction methods for foreground object segmentation, especially on complex image datasets with high variations. Specifically, one new feature construction method that incorporates the subtree technique in GP is designed, which can construct multiple features simultaneously (called SubtMFC, Subtree Multiple Feature Construction). Moreover, a parsimony pressure technique is introduced to improve SubtMFC for bloat control (a common issue for GP-based methods), which forms the method, PSubtMFC (Parsimony SubtMFC). In addition, comparison of popular GP-based feature construction methods for foreground object segmentation is conducted for the first time. Results show that SubtMFC achieves better or similar performance compared with three reference methods. In addition, compared with SubtMFC that does not control bloat, PSubtMFC can significantly reduce the solution size while maintain similar performance in the segmentation accuracy. The GP-based feature construction framework is further extended for feature representation based knowledge transfer, which can handle the problem of the scare labelled training data. Moreover, after GP is thoroughly investigated on benchmark datasets with one type of foreground objects (i.e. the Weizmann horse dataset and Pascal aeroplane dataset), it is considered whether the GP methods can perform well on datasets containing multiple types of foreground objects. Compared with three other well-performing GP-based feature construction methods, the proposed method achieves better or comparable results for the given segmentation tasks. In addition, this paper thoroughly compares/analyses popular GP-based feature construction methods for complex figure-ground segmentation for the first time. Moreover, further analyses on the input features frequently used by the GP-evolved feature construction functions reflect the effectiveness of the extracted high-level features.

Weakly-Supervised Learning of Category-Specific 3D Object Shapes.

Category-specific 3D object shape models have greatly boosted the recent advances in object detection, recognition and segmentation. However, even the most advanced approach for learning 3D object shapes still requires heavy manual annotations on large-scale 2D images. Such annotations include object categories, object keypoints, and figure-ground segmentation for the instances in each image. In particular, annotating figure-ground segmentation is unbearably labor-intensive and time-consuming. To address this problem, this paper devotes to learn category-specific 3D shape models under weak supervision, where only object categories and keypoints are required to be manually annotated on the training 2D images. By exploring the underlying relationship between two tasks: object segmentation and category-specific 3D shape reconstruction, we propose a novel weakly-supervised learning framework to jointly address these two tasks and combine them to boost the final performance of the learned 3D shape models. Moreover, learning without using figure-ground segmentation leads to ambiguous solutions. To this end, we develop the confidence weighting schemes in the viewpoint estimation and 3D shape learning procedure. These schemes effectively reduce the confusion caused by the noisy data and thus increase the chances for recovering more reliable 3D object shapes. Comprehensive experiments on the challenging PASCAL VOC benchmark show that our framework achieves comparable performance with the state-of-the-art methods that use expensive manual segmentation-level annotations. In addition, our experiments also demonstrate that our 3D shape models improve object segmentation performance.

Unsupervised Meta-Learning of Figure-Ground Segmentation via Imitating Visual Effects

This paper presents a “learning to learn” approach to figureground image segmentation. By exploring webly-abundant images of specific visual effects, our method can effectively learn the visual-effect internal representations in an unsupervised manner and uses this knowledge to differentiate the figure from the ground in an image. Specifically, we formulate the meta-learning process as a compositional image editing task that learns to imitate a certain visual effect and derive the corresponding internal representation. Such a generative process can help instantiate the underlying figure-ground notion and enables the system to accomplish the intended image segmentation. Whereas existing generative methods are mostly tailored to image synthesis or style transfer, our approach offers a flexible learning mechanism to model a general concept of figure-ground segmentation from unorganized images that have no explicit pixel-level annotations. We validate our approach via extensive experiments on six datasets to demonstrate that the proposed model can be end-to-end trained without ground-truth pixel labeling yet outperforms the existing methods of unsupervised segmentation tasks.

Spatial suppression promotes rapid figure-ground segmentation of moving objects

Segregation of objects from their backgrounds is a fundamental visual function and one that is particularly effective when objects are in motion. Theoretically, suppressive center-surround mechanisms are well suited for accomplishing motion segregation. This longstanding hypothesis, however, has received limited empirical support. We report converging correlational and causal evidence that spatial suppression of background motion signals is critical for rapid segmentation of moving objects. Motion segregation ability is strongly predicted by both individual and stimulus-driven variations in spatial suppression strength. Moreover, aging-related superiority in perceiving background motion is associated with profound impairments in motion segregation. This segregation deficit is alleviated via perceptual learning, but only when motion segregation training also causes decreased sensitivity to background motion. We argue that perceptual insensitivity to large moving stimuli effectively implements background subtraction, which, in turn, enhances the visibility of moving objects and accounts for the observed link between spatial suppression and motion segregation.

Border ownership-dependent tilt aftereffect for shape defined by binocular disparity and motion parallax.

Discerning objects from their surrounds (i.e., figure-ground segmentation) in a way that guides adaptive behaviors is a fundamental task of the brain. Neurophysiological work has revealed a class of cells in the macaque visual cortex that may be ideally suited to support this neural computation: border ownership cells (Zhou H, Friedman HS, von der Heydt R. J Neurosci 20: 6594–6611, 2000). These orientation-tuned cells appear to respond conditionally to the borders of objects. A behavioral correlate supporting the existence of these cells in humans was demonstrated with two-dimensional luminance-defined objects (von der Heydt R, Macuda T, Qiu FT. J Opt Soc Am A Opt Image Sci Vis 22: 2222–2229, 2005). However, objects in our natural visual environments are often signaled by complex cues, such as motion and binocular disparity. Thus for border ownership systems to effectively support figure-ground segmentation and object depth ordering, they must have access to information from multiple depth cues with strict depth order selectivity. Here we measured in humans (of both sexes) border ownership-dependent tilt aftereffects after adaptation to figures defined by either motion parallax or binocular disparity. We find that both depth cues produce a tilt aftereffect that is selective for figure-ground depth order. Furthermore, we find that the effects of adaptation are transferable between cues, suggesting that these systems may combine depth cues to reduce uncertainty (Bülthoff HH, Mallot HA. J Opt Soc Am A 5: 1749–1758, 1988). These results suggest that border ownership mechanisms have strict depth order selectivity and access to multiple depth cues that are jointly encoded, providing compelling psychophysical support for their role in figure-ground segmentation in natural visual environments.NEW & NOTEWORTHY Figure-ground segmentation is a critical function that may be supported by “border ownership” neural systems that conditionally respond to object borders. We measured border ownership-dependent tilt aftereffects to figures defined by motion parallax or binocular disparity and found aftereffects for both cues. These effects were transferable between cues but selective for figure-ground depth order, suggesting that the neural systems supporting figure-ground segmentation have strict depth order selectivity and access to multiple depth cues that are jointly encoded.

Real-world size coding of solid objects, but not 2-D or 3-D images, in visual agnosia patients with bilateral ventral lesions

Patients with visual agnosia show severe deficits in recognizing two-dimensional (2-D) images of objects, despite the fact that early visual processes such as figure-ground segmentation, and stereopsis, are largely intact. Strikingly, however, these patients can nevertheless show a preservation in their ability to recognize real-world objects –a phenomenon known as the ‘real-object advantage’ (ROA) in agnosia. To uncover the mechanisms that support the ROA, patients were asked to identify objects whose size was congruent or incongruent with typical real-world size, presented in different display formats (real objects, 2-D and 3-D images). While recognition of images was extremely poor, real object recognition was surprisingly preserved, but only when physical size matched real-world size. Analogous display format and size manipulations did not influence the recognition of common geometric shapes that lacked real-world size associations. These neuropsychological data provide evidence for a surprising preservation of size-coding of real-world-sized tangible objects in patients for whom ventral contributions to image processing are severely disrupted. We propose that object size information is largely mediated by dorsal visual cortex and that this information, together with detailed representation of object shape which is also subserved by dorsal cortex, serve as the basis of the ROA.

The effects of figure-ground segmentation on non-linear visual evoked potentials

The effects of figure-ground segmentation on non-linear visual evoked potentials

Figure-ground segmentation based on class-independent shape priors

We propose a method to generate figure-ground segmentation by incorporating shape priors into the graph-cuts algorithm. Given an image, we first obtain a linear representation of an image and then apply directional chamfer matching to generate class-independent, nonparametric shape priors, which provide shape clues for the graph-cuts algorithm. We then enforce shape priors in a graph-cuts energy function to produce object segmentation. In contrast to previous segmentation methods, the proposed method shares shape knowledge for different semantic classes and does not require class-specific model training. Therefore, the approach obtains high-quality segmentation for objects. We experimentally validate that the proposed method outperforms previous approaches using the challenging PASCAL VOC 2010/2012 and Berkeley (BSD300) segmentation datasets.

Feature-Specific Organization of Feedback Pathways in Mouse Visual Cortex

Higher and lower cortical areas in the visual hierarchy are reciprocally connected [1]. Although much is known about how feedforward pathways shape receptive field properties of visual neurons, relatively little is known about the role of feedback pathways in visual processing. Feedback pathways are thought to carry top-down signals, including information about context (e.g., figure-ground segmentation and surround suppression) [2-5], and feedback has been demonstrated to sharpen orientation tuning of neurons in the primary visual cortex (V1) [6, 7]. However, the response characteristics of feedback neurons themselves and how feedback shapes V1 neurons' tuning for other features, such as spatial frequency (SF), remain largely unknown. Here, using a retrograde virus, targeted electrophysiological recordings, and optogenetic manipulations, we show that putatively feedback neurons in layer 5 (hereafter "L5 feedback") in higher visual areas, AL (anterolateral area) and PM (posteromedial area), display distinct visual properties in awake head-fixed mice. AL L5 feedback neurons prefer significantly lower SF (mean: 0.04 cycles per degree [cpd]) compared to PM L5 feedback neurons (0.15 cpd). Importantly, silencing AL L5 feedback reduced visual responses of V1 neurons preferringlow SF (mean change in firing rate: -8.0%), whereas silencing PM L5 feedback suppressed responses of high-SF-preferring V1 neurons (-20.4%). These findings suggest that feedback connections from higher visual areas convey distinctly tuned visual inputs to V1 that serve to boost V1 neurons' responses to SF. Such like-to-like functional organization may represent animportant feature of feedback pathways in sensorysystems and in the nervous system in general.

How perception guides action: Figure-ground segmentation modulates integration of context features into S-R episodes.

Perception and action are closely related. Responses are assumed to be represented in terms of their perceptual effects, allowing direct links between action and perception. In this regard, the integration of features of stimuli (S) and responses (R) into S-R bindings is a key mechanism for action control. Previous research focused on the integration of object features with response features while neglecting the context in which an object is perceived. In 3 experiments, we analyzed whether contextual features can also become integrated into S-R episodes. The data showed that a fundamental principle of visual perception, figure-ground segmentation, modulates the binding of contextual features. Only features belonging to the figure region of a context but not features forming the background were integrated with responses into S-R episodes, retrieval of which later on had an impact upon behavior. Our findings suggest that perception guides the selection of context features for integration with responses into S-R episodes. Results of our study have wide-ranging implications for an understanding of context effects in learning and behavior. (PsycINFO Database Record

Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization.

Segmentation of histopathology sections is a necessary preprocessing step for digital pathology. Due to the large variability of biological tissue, machine learning techniques have shown superior performance over conventional image processing methods. Here we present our deep neural network-based approach for segmentation and classification of glands in tissue of benign and malignant colorectal cancer, which was developed to participate in the GlaS@MICCAI2015 colon gland segmentation challenge. We use two distinct deep convolutional neural networks (CNN) for pixel-wise classification of Hematoxylin-Eosin stained images. While the first classifier separates glands from background, the second classifier identifies gland-separating structures. In a subsequent step, a figure-ground segmentation based on weighted total variation produces the final segmentation result by regularizing the CNN predictions. We present both quantitative and qualitative segmentation results on the recently released and publicly available Warwick-QU colon adenocarcinoma dataset associated with the GlaS@MICCAI2015 challenge and compare our approach to the simultaneously developed other approaches that participated in the same challenge. On two test sets, we demonstrate our segmentation performance and show that we achieve a tissue classification accuracy of 98% and 95%, making use of the inherent capability of our system to distinguish between benign and malignant tissue. Our results show that deep learning approaches can yield highly accurate and reproducible results for biomedical image analysis, with the potential to significantly improve the quality and speed of medical diagnoses.