Occlusion Boundaries Research Articles

Boundary matting for view synthesis

In the last few years, new view synthesis has emerged as an important application of 3D stereo reconstruction. While the quality of stereo has improved, it is still imperfect, and a unique depth is typically assigned to every pixel. This is problematic at object boundaries, where the pixel colors are mixtures of foreground and background colors. Interpolating views without explicitly accounting for this effect results in objects with a “cut-out” appearance. To produce seamless view interpolation, we propose a method called boundary matting, which represents each occlusion boundary as a 3D curve. We show how this method exploits multiple views to perform fully automatic alpha matting and to simultaneously refine stereo depths at the boundaries. The key to our approach is the 3D representation of occlusion boundaries estimated to sub-pixel accuracy. Starting from an initial estimate derived from stereo, we optimize the curve parameters and the foreground colors near the boundaries. Our objective function maximizes consistency with the input images, favors boundaries aligned with strong edges, and damps large perturbations of the curves. Experimental results suggest that this method enables high-quality view synthesis with reduced matting artifacts.

Spatial and spectral analysis of occluded motions

We present a spatio-temporal analysis of motion at occluding boundaries. The main result is an analytical description of the motions and the distortions that occur at the occluding boundary. Based on this result we analyze occluding motions in the Fourier domain and show that the distortion term has an hyperbolic decay independent of the shape of the occluding boundary. Moreover, we derive the exact expression for the distortion term for the case of straight boundaries. The results are illustrated by using simulations with synthetic movies.

A snake model for object tracking in natural sequences

Tracking moving objects in video sequences is a task that emerges in various fields of study: video analysis, computer vision, biomedical systems, etc. In the last decade, special attention has been drawn to problems concerning tracking in real-world environments, where moving objects do not obey any afore-known constraints about their nature and motion or the scenes they are moving in. Apart from the existence of noise and environmental changes, many problems are also concerned, due to background texture, complicated object motion, and deformable and/or articulated objects, changing their shape while moving along time. Another phenomenon in natural sequences is the appearance of occlusions between different objects, whose handling requires motion information and, in some cases, additional constraints. In this work, we revisit one of the most known active contours, the Snakes, and we propose a motion-based utilization of it, aiming at successful handling of the previously mentioned problems. The use of the object motion history and first order statistical measurements of it, provide us with information for the extraction of uncertainty regions, a kind of shape prior knowledge w.r.t. the allowed object deformations. This constraining also makes the proposed method efficient, handling the trade-off between accuracy and computation complexity. The energy minimization is approximated by a force-based approach inside the extracted uncertainty regions, and the weights of the total snake energy function are automatically estimated as respective weights in the resulting evolution force. Finally, in order to handle background complexity and partial occlusion cases, we introduce two rules, according to which the moving object region is correctly separated from the background, whereas the occluded boundaries are estimated according to the object's expected shape. To verify the performance of the proposed method, some experimental results are included, concerning different cases of object tracking, indoors and outdoors, with rigid and deformable objects, noisy and textured backgrounds, as well as appearance of occlusions.

Multiple motion analysis: in spatial or in spectral domain?

In this paper, we compare the effects of multiple motions in spatial and spectral representations of an image sequence. We describe multiple motions in both domains and establish a comparison regarding their inherent properties when discretized. Though the spectral model provides us with an explicit description of both occlusion and transparency, it turns out that its resolution is very limited. We show that the spatial domain represented by the spatio-temporal derivatives has superior resolution properties and is thus more appropriate for the treatment of occlusion. We present an algorithm which based on an initial estimate of the number of motions uses the shift-and-subtract technique to localize occlusion boundaries and to track their movement in occlusion sequences. The same technique is used to distinguish occlusion from transparency and to decompose transparency scenes into multi-layers.

Correspondence with cumulative similarity transforms

A local image transform based on cumulative similarity measures is defined and is shown to enable efficient correspondence and tracking near occluding boundaries. Unlike traditional methods, this transform allows correspondences to be found when the only contrast present is the occluding boundary itself and when the sign of contrast along the boundary is possibly reversed. The transform is based on the idea of a cumulative similarity measure which characterizes the shape of local image homogeneity; both the value of an image at a particular point and the shape of the region with locally similar and connected values is captured. This representation is insensitive to structure beyond an occluding boundary but is sensitive to the shape of the boundary itself, which is often an important cue. We show results comparing this method to traditional least-squares and robust correspondence matching.

Occlusion and the Interpretation of Visual Motion: Perceptual and Neuronal Effects of Context

Visual motion can be represented in terms of the dynamic visual features in the retinal image or in terms of the moving surfaces in the environment that give rise to these features. For natural images, the two types of representation are necessarily quite different because many moving features are only spuriously related to the motion of surfaces in the visual scene. Such "extrinsic" features arise at occlusion boundaries and may be detected by virtue of the depth-ordering cues that exist at those boundaries. Although a number of studies have provided evidence of the impact of depth ordering on the perception of visual motion, few attempts have been made to identify the neuronal substrate of this interaction. To address this issue, we devised a simple contextual manipulation that decouples surface motion from the motions of visual image features. By altering the depth ordering between a moving pattern and abutting static regions, the perceived direction of motion changes dramatically while image motion remains constant. When stimulated with these displays, many neurons in the primate middle temporal visual area (area MT) represent the implied surface motion rather than the motion of retinal image features. These neurons thus use contextual depth-ordering information to achieve a representation of the visual scene consistent with perceptual experience.

Design and Use of Linear Models for Image Motion Analysis

Linear parameterized models of optical flow, particularly affine models, have become widespread in image motion analysis. The linear model coefficients are straightforward to estimate, and they provide reliable estimates of the optical flow of smooth surfaces. Here we explore the use of parameterized motion models that represent much more varied and complex motions. Our goals are threefold: to construct linear bases for complex motion phenomenas to estimate the coefficients of these linear modelss and to recognize or classify image motions from the estimated coefficients. We consider two broad classes of motions: i) generic “motion features” such as motion discontinuities and moving barss and ii) non-rigid, object-specific, motions such as the motion of human mouths. For motion features we construct a basis of steerable flow fields that approximate the motion features. For object-specific motions we construct basis flow fields from example motions using principal component analysis. In both cases, the model coefficients can be estimated directly from spatiotemporal image derivatives with a robust, multi-resolution scheme. Finally, we show how these model coefficients can be use to detect and recognize specific motions such as occlusion boundaries and facial expressions.

Numerical 3-D Shape Inference from Image Brightness for Non-Lambertian Surface.

In this paper, we present an iterative algorithm that can be used to recover shape from shading for non-Lambertian surfaces using single incident light. In the method, the parameters determining the general reflectance map are identified using occluding boundary information. The normal of the object surface is estimated subsequently, and the 3-D shape is recovered. The experimental results show the effectiveness of our method for shape from shading.

Occlusion-adaptive, content-based mesh design and forward tracking

Two-dimensional (2-D) mesh-based motion compensation preserves neighboring relations (through connectivity of the mesh) as well as allowing warping transformations between pairs of frames; thus, it effectively eliminates blocking artifacts that are common in motion compensation by block matching. However, available 2-D mesh models, whether uniform or non-uniform, enforce connectivity everywhere within a frame, which is clearly not suitable across occlusion boundaries. To this effect, we hereby propose an occlusion-adaptive forward-tracking mesh model, where connectivity of the mesh elements (patches) across covered and uncovered region boundaries are broken. This is achieved by allowing no node points within the background to be covered (BTBC) and refining the mesh structure within the model failure (MF) region(s) at each frame. The proposed content-based mesh structure enables better rendition of the motion (compared to a uniform or a hierarchical mesh), while tracking is necessary to avoid transmission of all node locations at each frame. Experimental results show successful motion compensation and tracking.

3-D Shape Recovery from Image Brightness for non-Lambertian Surface

This report presents a method for recovery of shape from shading in the case of a non-Lambertian surface illuminated by only a single light source. In the method, we derive simultaneous equations that should be satisfied by parameters determining reflectance map. After identifying the parameters by solving these equations, the surface normal is estimated by the iterative method using the surface normal information on the occluding boundary, and 3-D shape is recovered. The experimental results show the effectiveness of our method for shape from shading.

A Theory of Specular Surface Geometry

A theoretical framework is introduced for the perception of specular surface geometry. When an observer moves in three-dimensional space, real scene features such as surface markings remain stationary with respect to the surfaces they belong to. In contrast, a virtual feature which is the specular reflection of a real feature, travels on the surface. Based on the notion of caustics, a feature classification algorithm is developed that distinguishes real and virtual features from their image trajectories that result from observer motion. Next, using support functions of curves, a closed-form relation is derived between the image trajectory of a virtual feature and the geometry of the specular surface it travels on. It is shown that, in the 2D case, where camera motion and the surface profile are coplanar, the profile is uniquely recovered by tracking just two unknown virtual features. Finally, these results are generalized to the case of arbitrary 3D surface profiles that are traveled by virtual features when camera motion is not confined to a plane. This generalization includes a number of mathematical results that substantially enhance the present understanding of specular surface geometry. An algorithm is developed that uniquely recovers 3D surface profiles using a single virtual feature tracked from the occluding boundary of the object. All theoretical derivations and proposed algorithms are substantiated by experiments.

Numerical 3-D Shape Inference from Image Brightness for Non-Lambertian Surface.

In this paper, we present an iterative algorithm that can be used to recover shape from shading for non-Lambertian surfaces using single incident light. In the method, the parameters determining the general reflectance map are identified using occluding boundary information. The normal of the object surface is estimated subsequently, and the 3-D shape is recovered. The experimental results show the effectiveness of our method for shape from shading.

Occlusion edge blur: a cue to relative visual depth.

We studied whether the blur/sharpness of an occlusion boundary between a sharply focused surface and a blurred surface is used as a relative depth cue. Observers judged relative depth in pairs of images that differed only in the blurriness of the common boundary between two adjoining texture regions, one blurred and one sharply focused. Two experiments were conducted; in both, observers consistently used the blur of the boundary as a cue to relative depth. However, the strength of the cue, relative to other cues, varied across observers. The occlusion edge blur cue can resolve the near/far ambiguity inherent in depth-from-focus computations.

Displacement field estimation and image segmentation using block matching enhanced by a neural network.

The block-matching method plays an important role in displacement field estimation due to its simplicity, achievement of long-range motion, and robustness to noise. In this paper, a single-layer feedback neural network model is proposed that enhances block matching, estimates the displacement field, and simultaneously performs image segmentation from consecutive images. In this paper, image segmentation is defined as partitioning each image into a set of moving objects and the background. For any two consecutive images, a neural network is created that learns the connection relationship of the pixels in an object from the displacement field and stores the relationship in the network. A modified block matching is used to compute a more accurate displacement field by utilizing the segmentation information embedded in the neural network. The displacement vector at the edge of an object or occluding boundary is hard to estimate, but the proposed model performs satisfactorily because it learns and uses the connection information. Furthermore, a flood-fill algorithm is used to compute the dense displacement field more efficiently and correctly than the exhaustive search does. The most important aspect of this paper is that image segmentation is performed simultaneously with the displacement-field estimation by the neural-network model. The novel idea of the work is to embed the segmentation information (connection relations) in the neural network and to perform the displacement-field estimation and image segmentation simultaneously. Two methods for retrieving segmentation information from the neural network with any two consecutive images are also presented.

宇宙ロボット 正規化した明度情報を利用したオクルージョン境界のステレオ対応付け

宇宙ロボット 正規化した明度情報を利用したオクルージョン境界のステレオ対応付け

Perception of rigid motion in depth from the optical deformations of shadows and occlusion boundaries.

Perception of rigid motion in depth from the optical deformations of shadows and occlusion boundaries.

Perception of rigid motion in depth from the optical deformations of shadows and occlusion boundaries.

Three experiments were designed to examine the abilities of observers to determine an object's 3-dimensional structure and motion from various types of optical deformations. Observers were required to discriminate whether pairs of moving ellipsoids were rotating rigidly about a single axis or nonrigidly about different axes that varied in slant. Discrimination thresholds were significantly influenced by whether the ellipsoids were intersecting or nonintersecting and whether they contained identifiable texture elements. Performance was unaffected by precession movements of the axis of rotation, by increasing the number of intersecting ellipsoids beyond 2, or by replacing the deforming silhouettes with the projected motions of cast shadows presented in isolation against a planar background. These findings indicate that observers can perceive structure from motion based on several different types of optical deformation, including the deformations of shadows and silhouettes that do not contain identifiable features on which most existing theoretical analyses are designed to operate.

The minimal effect of occlusion on perceived depth from motion parallax

Two experiments tested the ability of subjects to perceive the depths of three-dimensional (3-D) objects oscillating behind a series of occluding bars. Occlusion was manipulated by the spatial period of the occluding bar series as well as by its duty cycle (the proportion of field of view that was not occluded). It was found that the estimates of perceived 3-D shape from motion parallax were remarkably unaffected when the projected image of a rotating 3-D object was partially occluded. Occlusion of 60% of the object had little effect on perceived shape, as judged by estimates of perceived depth, and even occlusion of 90% of the object produced only slight reductions in perceived depth in many cases. It was also found that even when there were no obvious occluding boundaries, judgments of depth were relatively unaffected.

Comment on “Numerical shape from shading and occluding boundaries”

The paper “Numerical shape from shading and occluding boundaries” [10] proposed a method to determine the 3D shape of an object from a single brightness image using the smoothness constraint on surface orientations, and the orientations of the surface given along an occluding boundary. This paper contributed both a specific algorithm for solving a specific problem, and a general approach that could be applied in other low-level vision problems.

Numerical 3-D Shape Inference from Image Brightness

For the purpose of 3-dimensional (3-D) shape inference from image brightness, it is necessary to establish the direction of light. In this paper, we propose an iterative algorithm for estimating the direction of light from the brightness and the surface normal on the occluding boundary. From the numerical results, it is found that the algorithm yields a reasonable solution in a small number of iterations. Furthermore, we consider the convergence of the 3-D shape inference algorithm, and show that the smaller the difference between the viewing direction and the direction of light, the faster the algorithm converges.