Stereo Disparity Research Articles

Unification scheme for 3D surface reconstruction using physically based models

AbstractWe propose a unification framework for three‐dimensional shape reconstruction using physically based models. A variety of 3D shape reconstruction techniques have been developed in the past two decades, such as shape from stereopsis, from shading, from texture gradient, and from structured lighting. However, the lack of a general theory that unifies these shape reconstruction techniques into one framework hinders the effort of a synergistical image interpretation scheme using multiple sensors/information sources. Most shape‐from‐X techniques use an “observable” (e.g., the stereo disparity, intensity, or texture gradient) and a model, which is based on specific domain knowledge (e.g., the triangulation principle, reflectance function, or texture distortion equation) to predict the observable, in 3D shape reconstruction. We show that all these “observable–prediction‐model” types of techniques can be incorporated into our framework of energy constraint on a flexible, deformable image frame. In our algorithm, if the observable does not confirm to the predictions obtained using the corresponding model, a large “error” potential results. The error potential gradient forces the flexible image frame to deform in space. The deformation brings the flexible image frame to “wrap” onto the surface of the imaged 3D object. Surface reconstruction is thus achieved through a “package wrapping” or a “shape deformation” process by minimizing the discrepancy in the observable and the model prediction. The dynamics of such a wrapping process are governed by the least action principle which is physically correct. A physically based model is essential in this general shape reconstruction framework because of its capability to recover the desired 3D shape, to provide an animation sequence of the reconstruction, and to include the regularization principle into the theory of surface reconstruction.

Serial Search for Targets Defined by Divergence or Deformation of Optic Flow

The optic flow field can be described in terms of the local differential measures, divergence, deformation, and rotation, which are informative about observer motion and the 3-D structure of the environment. Does an explicit representation of these measures exist in human visual processing in the form of a feature map? Triesman's criteria were used to investigate this; ie is there 'pop-out' for a target defined as different in local divergence or deformation from surrounding elements, or is a serial search necessary? The stimulus arrays contained 3, 5, or 9 square or rectangular elements, which each underwent repeated cycles of expansion, contraction, or deformation. The time required to detect a target undergoing the opposite transformation increased steeply with the number of elements, implying very slow serial search. (The mean time was 210 ms per element for divergence targets and 542 ms per element for deformation). The process was clearly still serial when the density and number of elements was increased up to 48 in an array 2.16 deg x 2.16 deg. In contrast, a single line element undergoing the opposite direction of translation motion to the rest of the display did show pop-out. It is concluded that no parallel processes seem to exist which are sensitive to the spatial uniformity of divergence and of deformation of optic flow. These differential properties may be derived as conjunctions of signals from a primary process which extracts local velocity. This result contrasts with our findings for targets defined by stereo disparity gradient, which show parallel processing in analogous experiments.

A cost-benefit analysis of a third camera for stereo correspondence

This paper looks at the twin issues of the gain in accuracy of stereo correspondence and the accompanying increase in computational cost due to the use of a third camera for stereo analysis. Trinocular stereo algorithms differ from binocular algorithms essentially in the epipolar constraint used in the local matching stage. The current literature does not provide any insight into the relative merits of binocular and trinocular stereo matching with the matching accuracy being verified aginst the ground truth. Experiments for evaluating the relative performance of binocular and trinocular stereo algorithms were conducted. The stereo images used for the performance evaluation were generated by applying a Lambertian reflectance model to real Digital Elevation Maps (DEMs) available from the U.S. Geological Survey. The matching accuracy of the stereo algorithms was evaluated by comparing the observed stereo disparity against the ground truth derived from the DEMs. It was observed that trinocular local matching reduced the percentage of mismatches having large disparity errors by more than half when compared to binocular matching. On the other hand, trinocular stereopsis increased the computational cost of local matching over binocular by only about one-fourth. We also present a quantization-error analysis of the depth reconstruction process for the nonparallel stereo-imaging geometry used in our experiments.

Erratum for Lehky et al., Neural Model of Stereoacuity and Depth Interpolation Based on a Distributed Representation of Stereo Disparity

S. R. Lehky and T. J. Sejnowski, the authors of “Neural Model of Stereoacuity and Depth Interpolation Based on a Distributed Representation of Stereo Disparity” (The Journal of Neuroscience,vol. 10 no. 7, July 1990, pp. 2281–2299) would like to acknowledge some errors in their published equations, although the results in the paper were obtained using the correct equations and therefore are not affected by the corrections. Equation 3 should read as follows:Given this number, the value ofpi, is given by:[See equation in the PDF file]The threshold criterion parameter was set to 0.50 rather than 0.75 correct discrimination. Equation 9 should read:[See equation in the PDF file]Mechanismi: i= 1,17

Direct recovery of three-dimensional scene geometry from binocular stereo disparity

An analysis of disparity is presented. It makes explicit the geometric relations between a stereo disparity field and a differentially project scene. These results show how it is possible to recover three-dimensional surface geometry through first-order (i.e., distance and orientation of a surface relative to an observer) and binocular viewing parameters in a direct fashion from stereo disparity. As applications of the analysis, algorithms have been developed for recovering three-dimensional surface orientation and discontinuities from stereo disparity. The results of applying these algorithms to natural image binocular stereo disparity information are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Cooperative vision integration through data-parallel neural computations

The authors describe a neural network approach for combining processing of multiple early vision modules. Energy functions for coupling the computation of intensity contours, optical flow, and stereo disparity are defined. Hopfield neural networks are used for function minimization with deterministic annealing to avoid spurious local minima. Vision integration schemes are developed by extending the work of T.A. Poggio et al. (1988) to include cooperative interactions between different vision modules and the Hebbian adaptation of vision module coupling on a massively parallel computer consisting of 4096 processing elements operated in a single-instruction-multiple-data mode. Simple experiments assess the performance of various integration approaches. The resulting algorithms facilitate fast, robust image segmentation.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Vertical disparities and perception of three-dimensional shape

The information about depth and three-dimensional shape available from the horizontal component of the stereo disparity field requires interpretation in conjunction with information about egocentric viewing distance (D). A novel computational approach for estimating D was proposed by Mayhew and Longuet-Higgins, who demonstrated that the horizontal gradient of vertical disparities uniquely specifies the viewing distance. We have now used random dot stereograms in a shape judgement task to show that changes in vertical disparities have no effect on perceived three-dimensional shape. Changes in ocular convergence do alter perceived shape, suggesting substantial changes in the subjects' scaling of horizontal disparities. We conclude that vertical disparities are not used to scale disparities for viewing distance, and that extraretinal signals must be considered when analysing human three-dimensional shape perception.

Neural model of stereoacuity and depth interpolation based on a distributed representation of stereo disparity [published erratum appears in J Neurosci 1991 Mar;11(3):following Table of Contents

We have developed a model for the representation of stereo disparity by a population of neurons that is based on tuning curves similar in shape to those measured physiologically (Poggio and Fischer, 1977). Signal detection analysis was applied to the model to generate predictions of depth discrimination thresholds. Agreement between the model and human psychophysical data was possible in this model only when the population size representing disparity in a small patch of visual field was in the range of about 20-200 units. Interval encoding and rate encoding were found to be inconsistent with these data. Psychophysical data on stereo interpolation (Westheimer, 1986a) suggest that there are short-range excitatory and long-range inhibitory interactions between disparity-tuned units at nearby spatial locations. We extended our population model of disparity coding at a single spatial location to include such lateral interactions. When there was a small disparity gradient between stimuli at 2 locations, units in the intermediate, unstimulated position developed a pattern of activity corresponding to the average of the 2 lateral disparities. When there was a large disparity gradient, units at the intermediate position developed a pattern of activity corresponding to an independent superposition of the 2 lateral disparities, so that both disparities were represented simultaneously. This mixed population pattern may underlie the perception of depth discontinuities and transparent surfaces. Similar types of distributed representations may be applicable to other parameters, such as orientation, motion, stimulus size, and motor coordinates.

Stereo and Motion Cues in Preattentive Vision Processing—Some Experiments with Random-Dot Stereographic Image Sequences

Low-level preattentive vision processing is of special interest since it seems the logical starting point of all vision processing. Exploration of the human visual processing system at this level is, however, extremely difficult, but can be facilitated by the use of stroboscopic presentation of sequences of random-dot stereograms, which contain only local spatial and temporal information and therefore limit the processing of these images to the low level. Four experiments are described in which such sequences were used to explore the relationships between various cues (optical flow, stereo disparity, and accretion and deletion of image points) at the low level. To study these relationships in more depth, especially the resolution of conflicting information among the cues, some of the image sequences presented information not usually encountered in 'natural' scenes. The results indicate that the processing of these cues is undertaken as a set of cooperative processes.

Qualitative depth from stereo, with applications

Obtaining exact depth from binocular disparities is hard if camera calibration is needed. We will show that qualitative information can be obtained from stereo disparities with little computation and without prior knowledge (or computation) of camera parameters. First, we derive two expressions that order all matched points in the images by depth in two distinct ways from image coordinates only. Using one for tilt estimation and point separation (in depth) demonstrates some anomalies observed in psychophysical experiments, most notably the “induced size effect.” We apply the same approach to detect qualitative changes in the curvature of a contour on the surface of an object, with either x- or y-coordinate fixed. Second, we develop an algorithm to compute axes of zero-curvature from disparities alone. The algorithm is shown to be quite robust against violations of its basic assumptions for synthetic data with relatively large controlled deviations. It performs almost as well on real images, as demonstrated on an image of four cans at different orientations.

Stereo ranging with verging cameras

A method of computing absolute range from stereo disparities by using verging cameras is presented. The approach differs from others by concentrating, through both analysis and experiment, on the effects caused by convergence, rather than on the general camera calibration problem. To compute stereo disparities, linear image features are extracted and matched using a hypothesize-and-verify method. To compute range, the relationship between object distance, vergence angle, and disparity is derived. Experimental results show the precision of the range computation, excluding mistaken matches, to be approximately 5% for object distances up to three meters and a baseline distance of 13 cm. Including mistaken matches results in performance that is an order of magnitude worse, leading the authors to suggest methods to identify and model them.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Neural Models of Binocular Depth Perception

We have developed a model for the representation of stereo disparity by using a population of neurons that is based on tuning curves similar in shape to those measured physiologically (Lehky and Sejnowski 1990). The model predicts depth discrimination thresholds that agree with human psychophysical data only when the population size representing disparity in a small patch of visual field was in the range of about 20-200 units. This population model of disparity coding at a single spatial location was extended to include lateral interactions, as suggested by psychophysical data on stereo interpolation (Westheimer 1986a). Disparity is a measure of depth relative to the plane of fixation. Additional sources of information are needed to estimate the distance to the point of fixation, such as that provided by eye vergence, vertical disparity, and accommodation. We have developed a simple model that combines disparity information in a distributed representation and vergence information to compute the absolute depth of objects from the observer (Pouget and Sejnowski 1990). Although these are models of binocular vision, a number of the ideas presented here generalize to the representations of other sensory cues.

Vision system for three-dimensional position measurement based on stereo disparity

A stereo disparity-based range finding technique was developed for measuring three-dimensional coordinates of object points. It used an image-matching algorithm which functionally consisted of scene reduction, epipolar line selection, and features matching. Experiments implementing this technique were performed on an industrial vision system. An error analysis indicated that image resolution was the major source of measurement inaccuracy.

A regularization model for stereo vision with controlled continuity

The problem of the computation of stereo disparity is approached as a mathematically ill-posed problem by using regularization theory. A controlled continuity constraint which provides a local spatial control over the smoothness of the solution enables the problem to be regularized while preserving the disparity discontinuities. The discontinuities are localized during the regularization process by examining the size of the disparity gradient at the gray value edges. An iterative algorithm for the computation of stereo disparity is obtained, and a computer experiment with synthetic data is shown.

Neural integration of information specifying structure from stereopsis and motion.

When one views a two-dimensional parallel projection of dots on the surface of a rotating globe, the direction of rotation is ambiguous, and the perceived direction of rotation of the two-dimensional figure is unstable over time. Stability can be temporarily induced by adaptation to a three-dimensional globe with a direction of rotation unambiguously specified by stereo disparity; adaptation causes the two-dimensional figure to appear to rotate in the direction opposite that experienced during stereoscopic adaptation. This adaptation effect is selective for axis of rotation but is not shape-specific. It does depend on simultaneous stimulation by multiple depth planes defined by elements moving in different directions. Evidently information about stereopsis and information about structure from motion are integrated within a common neural site in the brain.

Computation of stereo disparity using regularization

The computation of stereo disparity is a mathematically ill-posed problem. However, using regularization theory it may be transformed into a well-posed problem. Standard regularization can be employed to solve ill-posed problems by using stabilizing functionals that impose global smoothness constraints on acceptable solutions. However, the presence of depth discontinuities causes serious difficulties in standard regularization, since smoothness assumptions do not hold across discontinuities. This paper presents a regularization approach to stereopsis based on controlled-continuity stabilizing functionals. These functionals provides a spatial control over smoothness, allowing the introduction of discontinuities into the solution. An iterative method for the computation of stereo disparity is derived, and the result of a computer simulation with a synthetic stereo pair of images is shown.

Stereo disparity computation using Gabor filters

A solution to the correspondence problem for stereopsis is proposed using the differences in the complex phase of local spatial frequency components. One-dimensional spatial Gabor filters (Gabor 1946; Marcelja 1980), at different positions and spatial frequencies are convolved with each member of a stereo pair. The difference between the complex phase at corresponding points in the two images is used to find the stereo disparity. Disparity values are combined across spatial frequencies for each image location. Three-dimensional depth maps have been computed from real images under standard lighting conditions, as well as from random-dot stereograms (Julesz 1971). The algorithm can discriminate disparities significantly smaller than the width of a pixel. It is possible that a similar mechanism might be used in the human visual system.

Probing depth in monocular images.

It is generally expected that depth (distance) is the internal representational primitive that corresponds to much of the perception of 3D. We tested this assumption in monocular surface stimuli that are devoid of distance information (due to orthographic projection and the chosen surface shape, with perspective projection used as a control) and yet are vividly three-dimensional. Slant judgments were found to be in close correspondence with the actual geometric slant of the stimuli; the spatial orientation of the surfaces was perceived accurately. The apparent depth in these stimuli was then tested by superimposing a stereo depth probe over the monocular surface. In both the perspective and orthographic projection the gradient of perceived depth, measured by matching the apparent depth of the stereo probe with that of the monocular surface at a series of locations, was substantial. The experiments demonstrate that in orthographic projection the visual system can compute from local surface orientation a depth quantity that is commensurate with the relative depth derived from stereo disparity. The depth data suggests that, at least in the near field, the zero value for relative depth lies at the same absolute depth as the stereo horopter (locus of zero stereo disparity). Relative to this zero value, the depth-from-slant computation seems to provide an estimate of distance information that is independent of the absolute distance to the surface.

A New Sense for Depth of Field

This paper examines a novel source of depth information: focal gradients resulting from the limited depth of field inherent in most optical systems. Previously, autofocus schemes have used depth of field to measured depth by searching for the lens setting that gives the best focus, repeating this search separately for each image point. This search is unnecessary, for there is a smooth gradient of focus as a function of depth. By measuring the amount of defocus, therefore, we can estimate depth simultaneously at all points, using only one or two images. It is proved that this source of information can be used to make reliable depth maps of useful accuracy with relatively minimal computation. Experiments with realistic imagery show that measurement of these optical gradients can provide depth information roughly comparable to stereo disparity or motion parallax, while avoiding image-to-image matching problems.

Binocular Image Flows: Steps Toward Stereo-Motion Fusion

The analyses of visual data by stereo and motion modules have typically been treated as separate parallel processes which both feed a common viewer-centered 2.5-D sketch of the scene. When acting separately, stereo and motion analyses are subject to certain inherent difficulties; stereo must resolve a combinatorial correspondence problem and is further complicated by the presence of occluding boundaries, motion analysis involves the solution of nonlinear equations and yields a 3-D interpretation specified up to an undetermined scale factor. A new module is described here which unifies stereo and motion analysis in a manner in which each helps to overcome the other's short-comings. One important result is a correlation between relative image flow (i.e., binocular difference flow) and stereo disparity; it points to the importance of the ratio ¿ ¿, rate of change of disparity ¿ to disparity ¿, and its possible role in establishing stereo correspondence. The importance of such ratios was first pointed out by Richards [19]. Our formulation may reflect the human perception channel probed by Regan and Beverley [18].