Contour Junctions Research Articles

The mid-level vision toolbox for computing structural properties of real-world images

Mid-level vision is the intermediate visual processing stage for generating representations of shapes and partial geometries of objects. Our mechanistic understanding of these operations is limited, in part, by a lack of computational tools for analyzing image properties at these levels of representation. We introduce the Mid-Level Vision (MLV) Toolbox, an open-source software that automatically processes low- and mid-level contour features and perceptual grouping cues from real-world images. The MLV toolbox takes vectorized line drawings of scenes as input and extracts structural contour properties. We also include tools for contour detection and tracing for the automatic generation of vectorized line drawings from photographs. Various statistical properties of the contours are computed: the distributions of orientations, contour curvature, and contour lengths, as well as counts and types of contour junctions. The toolbox includes an efficient algorithm for computing the medial axis transform of contour drawings and photographs. Based on the medial axis transform, we compute several scores for local mirror symmetry, local parallelism, and local contour separation. All properties are summarized in histograms that can serve as input into statistical models to relate image properties to human behavioral measures, such as esthetic pleasure, memorability, affective processing, and scene categorization. In addition to measuring contour properties, we include functions for manipulating drawings by separating contours according to their statistical properties, randomly shifting contours, or rotating drawings behind a circular aperture. Finally, the MLV Toolbox offers visualization functions for contour orientations, lengths, curvature, junctions, and medial axis properties on computer-generated and artist-generated line drawings. We include artist-generated vectorized drawings of the Toronto Scenes image set, the International Affective Picture System, and the Snodgrass and Vanderwart object images, as well as automatically traced vectorized drawings of set architectural scenes and the Open Affective Standardized Image Set (OASIS).

Modified Abdominal Skin Resection: A Novel Approach to High-Definition Body Contouring of the Abdomen

Introduction: We present a novel abdominal contouring procedure, called the modified abdominal skin resection, designed to maximize elimination of skin redundancy while allowing for simultaneous high-definition liposuction of the entire abdominal skin flap. Technical details of the modified abdominal skin resection will be presented including inclusion and exclusion criteria, complications, and outcomes. Materials and Methods: Strategic limiting of abdominal skin undermining, preserving of any visible perforators, using ultrasound liposuction, and if necessary, administering Renuvion J plasma skin contraction are components of the presented high-definition abdominal contouring procedure. Limited undermining allows for high-definition liposuction of the abdomen to create muscle highlights and smooth contour junctions. Results: We present case studies demonstrating superior contour results when compared to traditional tummy tucks without muscle plication or liposuction alone. We present technical details of the modified abdominal skin resection, inclusion and exclusion criteria, complications, and outcomes. Discussion: The modified abdominal skin resection allows for high definition abdominoplasty outcomes. These results are superior to traditional abdominoplasties that are performed without muscle plication. This novel abdominoplasty procedure allows for aggressive removal of fat, as well as maximal elimination of skin redundancy and laxity. Inclusion and exclusion criteria established in this paper dicatate whether a patient is a good candidate for the modified abdominal skin resection. Conclusion: We present technical details required to complete the high definition abdominoplasty without muscle plication which allows for waistline snatching and creation of abdominal muscle highlights. High definition body contouring principles are applied to the modified abdominal skin resection to achieve superior body contouring results.

Mid-level feature contributions to category-specific gaze guidance.

Our research has previously shown that scene categories can be predicted from observers' eye movements when they view photographs of real-world scenes. The time course of category predictions reveals the differential influences of bottom-up and top-down information. Here we used these known differences to determine to what extent image features at different representational levels contribute toward guiding gaze in a category-specific manner. Participants viewed grayscale photographs and line drawings of real-world scenes while their gaze was tracked. Scene categories could be predicted from fixation density at all times over a 2-s time course in both photographs and line drawings. We replicated the shape of the prediction curve found previously, with an initial steep decrease in prediction accuracy from 300 to 500 ms, representing the contribution of bottom-up information, followed by a steady increase, representing top-down knowledge of category-specific information. We then computed the low-level features (luminance contrasts and orientation statistics), mid-level features (local symmetry and contour junctions), and Deep Gaze II output from the images, and used that information as a reference in our category predictions in order to assess their respective contributions to category-specific guidance of gaze. We observed that, as expected, low-level salience contributes mostly to the initial bottom-up peak of gaze guidance. Conversely, the mid-level features that describe scene structure (i.e., local symmetry and junctions) split their contributions between bottom-up and top-down attentional guidance, with symmetry contributing to both bottom-up and top-down guidance, while junctions play a more prominent role in the top-down guidance of gaze.

Spatial relationships between contours impact rapid scene classification.

Photographs and line drawings of natural scenes are easily classified even when the image is only briefly visible to the observer. Contour junctions and points of high curvature have been shown to be important for perceptual organization (Attneave, 1954; Biederman, 1987) and have been proposed to be influential in rapid scene classification (Walther & Shen, 2014). Here, we manipulate the junctions in images, either randomly translating them, or selectively removing or maintaining them. Observers were better at classifying images when the contours were randomly translated (disrupting the junctions) than when the junctions were randomly shifted (partially disrupting contour information). Moreover, observers were better at classifying a scene when shown only segments between junctions, than when shown only the junctions, with the middle segments removed. These results suggest that categorizing line drawings of real-world scenes does not solely rely on junction statistics. The spatial locations of the junctions are important, as well as their relationships with one another. Furthermore, the segments between junctions appear to facilitate scene classification, possibly due to their involvement in symmetry relationships with other contour segments.

Figure and ground: how the visual cortex integrates local cues for global organization.

Inferring figure-ground organization in two-dimensional images may require different complementary strategies. For isolated objects, it has been shown that mechanisms in visual cortex exploit the overall distribution of contours, but in images of cluttered scenes where the grouping of contours is not obvious, that strategy would fail. However, natural scenes contain local features, specifically contour junctions, that may contribute to the definition of object regions. To study the role of local features in the assignment of border ownership, we recorded single-cell activity from visual cortex in awake behaving Macaca mulatta. We tested configurations perceived as two overlapping figures in which T- and L-junctions depend on the direction of overlap, whereas the overall distribution of contours provides no valid information. While recording responses to the occluding contour, we varied direction of overlap and variably masked some of the critical contour features to determine their influences and their interactions. On average, most features influenced the responses consistently, producing either enhancement or suppression depending on border ownership. Different feature types could have opposite effects even at the same location. Features far from the receptive field produced effects as strong as near features and with the same short latency. Summation was highly nonlinear: any single feature produced more than two-thirds of the effect of all features together. These findings reveal fast and highly specific organization mechanisms, supporting a previously proposed model in which "grouping cells" integrate widely distributed edge signals with specific end-stopped signals to modulate the original edge signals by feedback. NEW & NOTEWORTHY Seeing objects seems effortless, but defining objects in a scene requires sophisticated neural mechanisms. For isolated objects, the visual cortex groups contours based on overall distribution, but this strategy does not work for cluttered scenes. Here, we demonstrate mechanisms that integrate local contour features like T- and L-junctions to resolve clutter. The process is fast, evaluates widely distributed features, and gives any single feature a decisive influence on figure-ground representation.

Contour junctions defined by dynamic image deformations enhance perceptual transparency.

The majority of work on the perception of transparency has focused on static images with luminance-defined contour junctions, but recent work has shown that dynamic image sequences with dynamic image deformations also provide information about transparency. The present study demonstrates that when part of a static image is dynamically deformed, contour junctions at which deforming and nondeforming contours are connected facilitate the deformation-based perception of a transparent layer. We found that the impression of a transparent layer was stronger when a dynamically deforming area was adjacent to static nondeforming areas than when presented alone. When contour junctions were not formed at the dynamic-static boundaries, however, the impression of a transparent layer was not facilitated by the presence of static surrounding areas. The effect of the deformation-defined junctions was attenuated when the spatial pattern of luminance contrast at the junctions was inconsistent with the perceived transparency related to luminance contrast, while the effect did not change when the spatial luminance pattern was consistent with it. In addition, the results showed that contour completions across the junctions were required for the perception of a transparent layer. These results indicate that deformation-defined junctions that involve contour completion between deforming and nondeforming regions enhance the perception of a transparent layer, and that the deformation-based perceptual transparency can be promoted by the simultaneous presence of appropriately configured luminance and contrast-other features that can also by themselves produce the sensation of perceiving transparency.

Training convolutional neural network from multi-domain contour images for 3D shape retrieval

Recent vision-brain physiological experiments [65] have demonstrated that contours, and in particular, contour junctions present in the 2D images are very informative for revealing the 3D structure of the object. Inspired by this observation, we take 2D sketches (or 2D views of 3D sketches) and edge maps of 2D views of 3D models as a unified domain to train the Convolutional Neural Network (CNN). The CNN features are then used for 3D object representation. We show that the CNN can successfully learn the object structure from different types of clues. The performance of the proposed method demonstrates that the semantic gap between the 2D/3D sketches and the 3D models can be bridged without any cross-domain similarity learning. Experiments show that our approach significantly outperforms the state-of-the-art 2D/3D sketch-based 3D retrieval methods.

Contour junctions underlie neural representations of scene categories in high-level human visual cortex

Humans efficiently grasp complex visual environments, making highly consistent judgments of entry-level category despite their high variability in visual appearance. How does the human brain arrive at the invariant neural representations underlying categorization of real-world environments? We here show that the neural representation of visual environments in scene-selective human visual cortex relies on statistics of contour junctions, which provide cues for the three-dimensional arrangement of surfaces in a scene. We manipulated line drawings of real-world environments such that statistics of contour orientations or junctions were disrupted. Manipulated and intact line drawings were presented to participants in an fMRI experiment. Scene categories were decoded from neural activity patterns in the parahippocampal place area (PPA), the occipital place area (OPA) and other visual brain regions. Disruption of junctions but not orientations led to a drastic decrease in decoding accuracy in the PPA and OPA, indicating the reliance of these areas on intact junction statistics. Accuracy of decoding from early visual cortex, on the other hand, was unaffected by either image manipulation. We further show that the correlation of error patterns between decoding from the scene-selective brain areas and behavioral experiments is contingent on intact contour junctions. Finally, a searchlight analysis exposes the reliance of visually active brain regions on different sets of contour properties. Statistics of contour length and curvature dominate neural representations of scene categories in early visual areas and contour junctions in high-level scene-selective brain regions.

Nonaccidental Properties Underlie Human Categorization of Complex Natural Scenes

Humans can categorize complex natural scenes quickly and accurately. Which scene properties enable people to do this with such apparent ease? We extracted structural properties of contours (orientation, length, curvature) and contour junctions (types and angles) from line drawings of natural scenes. All of these properties contain information about scene categories that can be exploited computationally. However, when we compared error patterns from computational scene categorization with those from a six-alternative forced-choice scene-categorization experiment, we found that only junctions and curvature made significant contributions to human behavior. To further test the critical role of these properties, we perturbed junctions in line drawings by randomly shifting contours and found a significant decrease in human categorization accuracy. We conclude that scene categorization by humans relies on curvature as well as the same nonaccidental junction properties used for object recognition. These properties correspond to the visual features represented in area V2.

Extraction of Surface-Related Features in a Recurrent Model of V1-V2 Interactions

BackgroundHumans can effortlessly segment surfaces and objects from two-dimensional (2D) images that are projections of the 3D world. The projection from 3D to 2D leads partially to occlusions of surfaces depending on their position in depth and on viewpoint. One way for the human visual system to infer monocular depth cues could be to extract and interpret occlusions. It has been suggested that the perception of contour junctions, in particular T-junctions, may be used as cue for occlusion of opaque surfaces. Furthermore, X-junctions could be used to signal occlusion of transparent surfaces.Methodology/Principal FindingsIn this contribution, we propose a neural model that suggests how surface-related cues for occlusion can be extracted from a 2D luminance image. The approach is based on feedforward and feedback mechanisms found in visual cortical areas V1 and V2. In a first step, contours are completed over time by generating groupings of like-oriented contrasts. Few iterations of feedforward and feedback processing lead to a stable representation of completed contours and at the same time to a suppression of image noise. In a second step, contour junctions are localized and read out from the distributed representation of boundary groupings. Moreover, surface-related junctions are made explicit such that they are evaluated to interact as to generate surface-segmentations in static images. In addition, we compare our extracted junction signals with a standard computer vision approach for junction detection to demonstrate that our approach outperforms simple feedforward computation-based approaches.Conclusions/SignificanceA model is proposed that uses feedforward and feedback mechanisms to combine contextually relevant features in order to generate consistent boundary groupings of surfaces. Perceptually important junction configurations are robustly extracted from neural representations to signal cues for occlusion and transparency. Unlike previous proposals which treat localized junction configurations as 2D image features, we link them to mechanisms of apparent surface segregation. As a consequence, we demonstrate how junctions can change their perceptual representation depending on the scene context and the spatial configuration of boundary fragments.

Tracking of Moving Objects in Video Through Invariant Features in Their Graph Representation

The paper suggests a contour-based algorithm for tracking moving objects in video. The inputs are segmented moving objects. Each segmented frame is transformed into region adjacency graphs (RAGs). The object's contour is divided into subcurves. Contour's junctions are derived. These junctions are the unique of the tracked object. Junctions from two consecutive frames are matched. The junctions' motion is estimated using RAG edges in consecutive frames. Each pair of matched junctions may be connected by several paths (edges) that become candidates that represent a tracked contour. These paths are obtained by the -shortest paths algorithm between two nodes. The RAG is transformed into a weighted directed graph. The final tracked contour construction is derived by a match between edges (subcurves) and candidate paths sets. The RAG constructs the tracked contour that enables an accurate and unique moving object representation. The algorithm tracks multiple objects, partially covered (occluded) objects, compounded object of merge/split such as players in a soccer game and tracking in a crowded area for surveillance applications. We assume that features of topologic signature of the tracked object stay invariant in two consecutive frames. The algorithm's complexity depends on RAG's edges and not on the image's size.

Effects of Partial Occlusion on Perceived Surface Segregation

Tests are reported of the possibility that local information from contour junctions and from corners of intersecting surfaces is used for perceived surface segregation. Stimuli were two intersecting squares with small disks occluding different parts of the squares. The perceived segregation of the squares from one another decreased as the amount of occlusion of parts of the squares increased. This segregation was less when disks occluded parts of contours than when disks occluded parts entirely inside the squares. Occlusion of parts of contours reduced segregation independently of whether contour junctions or corners were visible or invisible, both where the intersecting surfaces were transparent squares and when they were outlined squares. The present findings show that local information from contour junctions or from corners is not used for surface segregation, and confirm that this segregation is determined by global processes of grouping of areas and of extrapolation of contours.

The role of junctions in surface completion and contour matching.

It has been suggested that contour junctions may be used as cues for occlusion. Ecologically, T-junctions and L-junctions are concurrent with situations of occlusion: they arise when the bounding contour of the occluding surface intersects with that of the occluded surface. However, there are other image properties that can be used as cues for occlusion. Here the role of junctions is directly compared with other occlusion cues--specifically, relatability and surface-similarity--in the emergence of amodal completion and illusory contour perception. Stimuli have been constructed that differ only in the junction structure, with the other occlusion cues kept unchanged. L-junctions and T-junctions were eliminated from the image or manipulated so as to be locally inconsistent with the (still valid) global occlusion interpretation. Although the other occlusion cues of relatability and surface similarity still existed in the image, subjects reported not perceiving illusory contours or amodal completion in junction-manipulated images. Junction manipulation also affected the perceived stereoscopic depth and motion of image regions, depending on whether they were perceived to amodally complete with a disjoint region in the image. These results are interpreted in terms of the role of junctions in the processes of surface completion and contour matching. It is proposed that junctions, being a local cue for occlusion, are used to launch completion processes. Other, more global occlusion cues, such as relatability, play a part at a later stage, once completion processes have been launched.

Occlusion junctions do not improve stereoacuity.

Occlusion geometry gives rise to interocular shifts in the positions of binocularly viewed contour junctions. Since these shifts do not give rise to normal binocular disparities, they have been called 'pseudodisparities'. Previous work has shown that the unmatched contour segments of a partially occluded contour at occlusion junctions can be used to recover the geometry of the occluding surface through the construction of 'illusory' contours. Here, experiments were performed to determine whether such junction shifts could enhance stereoscopic depth detection when the relative disparity between the contours was below threshold. Our results showed that stereoscopic depth detection does not improve when pseudodisparity is present. We conclude that the visual system is less sensitive to pseudodisparity than to conventional disparity information. We suggest that the primary role of pseudodisparity is to overcome conditions of camouflage.

A theory of illusory lightness and transparency in monocular and binocular images: the role of contour junctions.

A theory of illusory transparency and lightness is described for monocular and binocular images containing X-, T- and I-contour junctions. This theory asserts that the geometric and luminance relationships of contour junctions induce illusory transparency and lightness percepts by causing a phenomenal scission of a homogenous luminance into multiple contributions. Specifically, it is argued that a discontinuous change in contrast along aligned contours that preserve contrast polarity induces a scission of the lower contrast region into a near-transparent surface or an illumination change, and a more distant surface that continues behind behind this near layer. This scission is assumed to cause changes in perceived lightness and/or surface opacity. Discontinuous changes in contrast along contours also are assumed to induce end-cut illusory contours that run roughly perpendicular to the inducing orientation of the contour, both monocularly and binocularly. Binocular illusory contours are shown to be caused by the presence of unmatchable contour terminators. It is argued that the presented theory can provide a unified account of a variety of monocular and binocular illusions that induce uniform transformations in perceived lightness, including neon-color spreading, the Munker-White illusion, Benary's illusion, and illusory monocular and binocular transparency.

Border Ownership as a Special Case of Contrast Ownership

The classification of boundaries as figure and ground is one of the fundamental computations performed by the visual system. This computation requires classifying one side of a boundary, as an occluding or ‘intrinsic’ contour, and the other side of the contour as occluded or ‘extrinsic’ (also described as the problem of border ownership). A recent analysis of stereoscopic contour junctions revealed that the classification of edges as intrinsic or extrinsic to a neighbouring surface depends critically on the relative depth of the contours at the image junction. Specifically, it was argued that near edges are intrinsically one-sided (ie border ownership is uniquely specified by the junction geometry), whereas far contours are inherently two-sided (ie the border ownership of the far contour cannot be determined on the basis of the local junction geometry). Here I present new stimuli that reveal that the asymmetry of near and far contours in specifying border ownership reflects a more general asymmetry in the classification of near and far contrasts. A variety of stimuli were created. In one set of patterns, an array of five circular discs was embedded on a homogeneous background (either white or black) and was filled with a texture. A uniform disparity was added to the textured region such that it appeared nearer than the outer edges of the disc. When the power spectrum of the texture was systematically varied as functions of 1/ fn, the texture did not appear as a solid surface. Rather, the discs containing the texture gradients were decomposed into two sources: a constant background colour of the discs, and transparent clouds that appeared to float in front of the discs. Remarkably, the apparent colour of the clouds and the discs could be transformed simply by changing the colour of the background of the regions that neighboured the discs; the texture itself was unchanged. Specifically, when the textured discs were placed on white backgrounds, the pattern appeared as white clouds hovering in front of dark discs; but when the textured discs were placed on black backgrounds, the percept was of black ‘smoke’ hovering in front of white discs. It is argued that these results reveal that the problem of border ownership is a special case of contrast ownership. The two-sided qualities of far contrasts determine the colour of the background, which in turn determines which side of a near gradient is considered to be more or less opaque (ie which side is more occlusive).

A theoretical analysis of illusory contour formation in stereopsis.

A theoretical analysis of how illusory contours are formed in untextured stereograms is presented. This analysis focuses on the role of contour junctions in initiating surface interpolation processes. More specifically, this theory asserts that the matching geometry of stereoscopic junctions that is, the pattern of matchable and unmatchable features present at the intersection of binocularly viewed contours initiates processes of illusory contour formation (“modal” completion) and the connection of partially occluded objects (“amodal” completion). The matching geometry of three forms of stereoscopic junctions that elicit percepts of illusory surfaces is derived under the assumption that only horizontal disparities are matched. This analysis reveals the presence of two distinct kinds of monocular features generated by binocular viewing. Experiments and demonstrations are presented that reveal that: 1) these monocular features play a critical role in modal and amodal completion in stereopsis; and 2) that the difference in these two kinds of monocular features is responsible for asymmetries in the perceptual stability of modally and amodally completed surfaces.

A theoretical analysis of illusory contour formation in stereopsis.

A theoretical analysis of how illusory contours are formed in untextured stereograms is presented. This analysis focuses on the role of contour junctions in initiating surface interpolation processes. More specifically, this theory asserts that the matching geometry of stereoscopic junctions that is, the pattern of matchable and unmatchable features present at the intersection of binocularly viewed contours initiates processes of illusory contour formation (“modal” completion) and the connection of partially occluded objects (“amodal” completion). The matching geometry of three forms of stereoscopic junctions that elicit percepts of illusory surfaces is derived under the assumption that only horizontal disparities are matched. This analysis reveals the presence of two distinct kinds of monocular features generated by binocular viewing. Experiments and demonstrations are presented that reveal that: 1) these monocular features play a critical role in modal and amodal completion in stereopsis; and 2) that the difference in these two kinds of monocular features is responsible for asymmetries in the perceptual stability of modally and amodally completed surfaces.