Binocular Matching Research Articles

Detecting binocular half-occlusions: empirical comparisons of five approaches

Binocular half-occlusion points are those that are visible in one of the two views provided by a binocular imaging system. Due to their importance in binocular matching as well as, subsequent interpretation tasks, a number of approaches have been developed for dealing with such points. In the current paper, we consider five methods that explicitly detect half-occlusions and report on a more uniform comparison than has previously been performed. Taking a disparity image and its associated match goodness image as input, we generate images that show the half-occluded points in the underlying scene. We quantitatively and qualitatively compare these methods under a variety of conditions.

Envelope size tuning for stereo-depth perception of small and large disparities

Stereopsis is the sense of depth derived from binocular disparities that are formed between targets that are matched between the two retinal images. Binocular matches for sustained stereopsis are based on similarity of orientation, spatial frequency and contrast of the two retinal images whereas matches for transient stereopsis depend on these parameters to a very limited extent. In this investigation we have tested the possibility that transient stereopsis forms matches between objects of similar overall size. The tuning of sustained and transient stereopsis to contrast-envelope size was investigated by presenting narrow-band Gabor targets of unequal size to the two eyes. Bandwidth for envelope-size tuning was estimated from the range of dichoptic size-differences over which stereo performance remained above chance level. An equal bandwidth of 2 octaves was found for the sustained and transient stereo systems when stimulated with parallel orientation Gabors that subtended a small disparity. Sustained-stereo performance with orthogonal carriers was reduced with large envelope sizes. Bandwidth of the transient stereo system increased to 3 octaves when tested with a larger disparity stimulus and it was independent of carrier orientation. Reducing the contrast of the larger-size Gabor improved transient-stereo performance from near chance (48–58%) to 85–95%. Thus the bandwidth for envelope-size tuning is much broader than indicated with equal physical contrast stimuli. The observed tuning to envelope size, while broad, is tighter than that observed for carrier spatial-frequency [Vis. Res. 38 (1998) 3057], carrier orientation [Vis. Res. 39 (1999) 2717] and contrast polarity [Vis. Res. 39 (1999) 4010] of the stimulus. Thus it would appear that envelope size and, to a greater extent, temporal synchrony of the dichoptic stimuli [Perception 24 (1995) 33] are the primary means for selecting matched binocular inputs for transient stereopsis.

Luminance spatial scale facilitates stereoscopic depth segmentation.

Are differences in luminance spatial frequency between surfaces that overlap in depth useful for surface segmentation? We examined this question, using a novel stimulus termed a dual-surface disparity grating. The dual-surface grating was made from Gabor micropatterns and consisted of two superimposed sinusoidal disparity gratings of identical disparity-modulation spatial frequency and orientation but of opposite spatial phase. Corrugation amplitude thresholds for discrimination of the orientation of the dual-surface grating were obtained as a function of the difference in Gabor (luminance) spatial frequency between the two surfaces. When the Gabor micropatterns on the two surfaces were identical in spatial frequency, thresholds were very high and in some instances impossible to obtain. However, with as little as a 1-octave difference in spatial frequency between the surfaces, thresholds fell sharply to near-asymptotic levels. The fall in thresholds paralleled a change in the appearance of the stimulus from one of irregular depth to stereo transparency. The most parsimonious explanation for this finding is that the introduction of a between-surface luminance spatial-frequency difference reduces the number of spurious cross-surface binocular matches, thus helping to reveal the three-dimensional structure of the stimulus.

Motion direction, speed and orientation in binocular matching.

The spatial differences between the images seen by the two eyes, called binocular disparities, can be used to recover the volumetric (three-dimensional) aspects of a scene. The computation of disparity depends upon the correct identification of corresponding features in the two images. Understanding what image features are used by the brain to solve this matching problem is one of the main issues in stereoscopic vision. Many cortical neurons in visual areas V1 (ref. 2), MT (refs 3, 4) and MST (refs 5, 6) that are tuned to binocular disparity are also tuned to orientation, motion direction and speed. Although psychophysical work has shown that motion direction can facilitate binocular matching, the psychophysical literature on the role of orientation is mixed, and it has been argued that speed differences are ineffective in aiding correspondence. Here we use a different psychophysical paradigm to show that the visual system uses similarities in orientation, motion direction and speed to achieve binocular correspondence. These results indicate that cells that multiplex orientation, motion direction, speed and binocular disparity may help to solve the binocular matching problem.

Color-specific depth mechanisms revealed by a color-contingent depth aftereffect

Models of stereoscopic depth perception for both natural and random-dot images have focused mainly on the matching of achromatic features of binocular images. Recently, a growing body of research has investigated whether chromatic features can also contribute to the construction of stereoscopic depth. Here we present experiments yielding color-contingent depth aftereffects comparable in magnitude to those measured after adaptation to achromatic stimuli as evidence of neural mechanisms tuned to both color and depth. Furthermore, we report that the locus of the combined processing of color and depth is likely to lie beyond the site of binocular matching.

Unconstrained stereoscopic matching of lines

The computation of horizontal binocular disparities used in stereoscopic depth perception depends upon the identification of corresponding features in the two retinal images. In principle, binocular matching is a two-dimensional problem that considers matches in all possible meridians. Normally, constraints such as end points or crossing points limit the direction and magnitude of matches. If matching is unconstrained, such as is the case with long lines, it is completely ambiguous. Under these conditions the default match will be determined by the operating range, or upper disparity limit, of matchable vertical and horizontal disparities. We computed the operating range of vertical matches for stereoscopic depth as a function of line orientation. Our results suggest that the two-dimensional operating range is anisotropic for vertical and horizontal disparity and that unconstrained matches are not based upon either epipolar geometry or nearest neighbor constraints, but rather the mean of disparity estimates within the operating range for binocular matches. This operating range can be extended vertically when matches are constrained by image primitives.

Depth aliasing by the transient-stereopsis system

A fundamental problem in stereo-processing is determining which images in the two eyes correspond to the same object. This problem is particularly pronounced with periodic stimuli where it is theoretically possible to binocularly match a given feature in one eye with any of the identical features in the other eye. One way to minimise the likelihood of the occurrence of such aliasing is to restrict the upper-disparity limit that a particular binocular cell can process to one-half of the spatial period to which the cell is sensitive. While such a restriction would not be a major problem for the sustained stereo-system (which processes small disparities) it would be for the transient system (which is capable of processing disparities as large as 10°). Large-field sinewave variations in luminance were used to compare the propensity of the sustained and transient systems to exhibit depth aliasing-that is to signal a depth sign that corresponds to a binocular match that is greater than the nearest-neighbour pairing. Results were that: depth aliasing was exhibited at short, but not at long durations; decreasing the disparity of the stimulus reduced the likelihood of depth-aliasing; and the critical disparity for this reduction in depth aliasing was dependent upon the spatial frequency of the stimulus, i.e. it was phase, not absolute disparity dependent. Based upon these results, we conclude that while the sustained system implements the half-cycle disparity-processing limit, the transient system does not.

Is the development of orientation selectivity instructed by activity?

Is the development of orientation selectivity in visual cortex instructed by the patterns of neural activity of input neurons? We review evidence as to the role of activity, review models of activity-instructed development, and discuss how these models can be tested. The models can explain the normal development of simple cells with binocularly matched orientation preferences, the effects of monocular deprivation and reverse suture on the orientation map, and the development of a full intracortical circuit sufficient to explain mature response properties including the contrast-invariance of orientation tuning. Existing experiments are consistent with the models, in that (a) selective blockade of ON-center ganglion cells, which will degrade or eliminate the information predicted to drive development of orientation selectivity, in fact prevents development of orientation selectivity; and (b) the spontaneous activities of inputs serving the two eyes are correlated in the lateral geniculate nucleus at appropriate developmental times, as was predicted to be required to achieve binocular matching of preferred orientations. However, definitive tests remain to be done to firmly establish the instructive rather than simply permissive role of activity and determine whether the retinotopically and center type-specific patterns of activity predicted by the models actually exist. We conclude by critically examining alternative scenarios for the development of orientation selectivity and maps, including the idea that maps are genetically prespecified.

Binocular Stereo from Grey-Scale Images

Grey-scale images consist of physical measurements of light. Scale-space theories have been developed to unconfound these measurements from the detector grid. In this framework, we look into the problem of binocular stereo. On a sufficiently large scale, a pixel carries information not only of the grey-value, but of the entire grey-value n-jet, i.e., derivatives up to order n. The subject of this paper is to show, in a general context, how the scale-space n-jet can be exploited for binocular matching. The analysis leads (under appropriate assumptions) to a direct determination of the local n-jet of the disparity field. The general result is an analysis which could be incorporated into many existing stereo algorithms to improve their use of the grey value data. In the computational scheme presented here, the estimations are strictly local, but based on image derivatives at a scale where the image structure is significant. This scale is automatically selected by minimising computational uncertainty. Results are shown as direct computations of surface normals on synthetic and real images.

Linear and nonlinear transparencies in binocular vision.

When the product of a vertical square-wave grating (contrast envelope) and a horizontal sinusoidal grating (carrier) are viewed binocularly with different disparity cues they can be perceived transparently at different depths. We found, however, that the transparency was asymmetric; it only occurred when the envelope was perceived to be the overlaying surface. When the same two signals were added, the percept of transparency was symmetrical; either signal could be seen in front of or behind the other at different depths. Differences between these multiplicative and additive signal combinations were examined in two experiments. In one, we measured disparity thresholds for transparency as a function of the spatial frequency of the envelope. In the other, we measured disparity discrimination thresholds. In both experiments the thresholds for the multiplicative condition, unlike the additive condition, showed distinct minima at low envelope frequencies. The different sensitivity curves found for multiplicative and additive signal combinations suggest that different processes mediated the disparity signal. The data are consistent with a two-channel model of binocular matching, with multiple depth cues represented at single retinal locations.

Stereoscopic transparency: a test for binocular vision's disambiguating power.

It has been suggested that to resolve ambiguities implicit in binocular perception of complex visual scenes, the brain adopts a continuity constraint assuming that disparities change smoothly with eccentricity. Stereoscopic transparency is characterized by abrupt changes of binocular disparity across retinal locations. The focus of the present study is how the brain uses the continuity constraint in the perception of stereoscopic transparency despite the presence of abrupt disparity changes. Observers viewed random-dot stereograms of overlapping transparent plane and cylindrical surfaces and had to distinguish between two orientations of the cylindrical surface under conditions of strictly controlled depth fixation. Surprisingly, maximal dot density of the transparent plane at which perception is still veridical dramatically decreases as depth separation between the surfaces grows. Persistence of this relationship, when binocular matching processes at each surface are separated to on and off brightness channels, suggests at least two stages in the underlying computation binocular matching and inter-surface interactions. We show that these phenomena cannot be accounted for by either higher severity of matching with high dot densities or the ability of the denser surface to pull vergence to its depth. We also measure contrast sensitivity and near-far symmetry of the underlying mechanism and propose a model of competitive interactions between dissimilar disparities.

Cortical dynamics of three-dimensional surface perception: Binocular and half-occluded scenic images

Previous models of stereopsis have concentrated on the task of binocularly matching left and right eye primitives uniquely. A disparity smoothness constraint is often invoked to limit the number of possible matches. These approaches neglect the fact that surface discontinuities are both abundant in natural everyday scenes, and provide a useful cue for scene segmentation. da Vinci stereopsis refers to the more general problem of dealing with surface discontinuities and their associated unmatched monocular regions within binocular scenes. This study develops a mathematical realization of a neural network theory of biological vision, called FACADE theory, that shows how early cortical stereopsis processes are related to later cortical processes of three-dimensional surface representation. The mathematical model demonstrates through computer simulation how the visual cortex may generate three-dimensional boundary segmentations and use them to control filling-in of three-dimensional surface properties in response to visual scenes. Model mechanisms correctly match disparate binocular regions while filling-in monocular regions with the correct depth within a binocularly viewed scene. This achievement required the introduction of a new multiscale binocular filter for stereo matching which clarifies how cortical complex cells match image contours of like contrast polarity, while pooling signals from opposite contrast polarities. The filter also suggests how false binocular matches and unmatched monocular cues are automatically handled across multiple spatial scales. Pooling of signals from even- and odd-symmetric simple cells at complex cells helps to eliminate spurious activity peaks in matchable signals. Later stages of cortical processing by the blob and interblob streams, including refined models of cooperative boundary grouping and reciprocal stream interactions between boundary and surface representations, are modeled to provide a complete simulation of the da Vinci stereopsis percept.

The direction of retinal motion facilitates binocular stereopsis.

Visual information from binocular disparity and from relative motion provide information about three-dimensional structure and layout of the world. Although the mechanisms that process these cues have typically been studied independently, there is now a substantial body of evidence that suggests that they interact in the visual pathway. This paper investigates one advantage of such an interaction: whether retinal motion can be used as a matching constraint in the binocular correspondence process. Stimuli that contained identical disparity and motion signals but which differed in their fine-scale correlation were created to establish whether the direction, or the speed, of motion could enhance performance in a psychophysical task in which binocular matching is a limiting factor. The results of these experiments provide clear evidence that different directions of motion, but not different speeds, are processed separately in stereopsis. The results fit well with properties of neurons early in the cortical visual pathway which are thought to be involved in determining local matches between features in the two eyes' images.

Human stereo matching is not restricted to epipolar lines

Computational approaches to stereo matching have often taken advantage of a geometric constraint which states that matching elements in the left and right eye images will always fall on “epipolar lines”. The use of this epipolar constraint reduces the search space from two dimensions to one, producing a tremendous saving in the computation time required to find the matching solution. Use of this constraint requires a precise knowledge of the relative horizontal, vertical and torsional positions of the two eyes, however, and this information may be unavailable in many situations. Experiments with dynamic random element stereograms reveal that human stereopsis can detect and identify the depth of matches over a range of both vertical and horizontal disparity. Observers were able to make accurate near/far depth discriminations when vertical disparity was as large as 45 arcmin, and were able to detect the presence of correlation over a slightly larger range. Thus, human binocular matching sensitivity is not strictly constrained to epipolar lines.

The dynamics of vertical vergence.

We measured the gain and phase of vertical vergence in response to disjunctive vertical oscillations of dichoptic textured displays. The texture elements were m-scaled to equate visibility over the area of the display and were aperiodic and varied in shape so as to avoid spurious binocular matches. The display subtended 65 degrees and oscillated through peak-to-peak amplitudes from 18 arc min to 4 degrees at frequencies from 0.05 to 2 Hz - larger ranges than used in previous investigations. The gain of vergence was near 1 when the stimulus oscillated at 18 arc min at a frequency of 0.1 Hz or less. As the amplitude of stimulus oscillation increased from 18 arc min to 4 degrees, vergence gain decreased at all frequencies, which is evidence of a nonlinearity. Gain declined with increasing stimulus frequency but was still about 0.5 at 2 Hz for an amplitude of 18 arc min. Phase lag increased from less than 10 degrees at a stimulus frequency of 0.05 Hz to between 100 degrees and 145 degrees at 2 Hz. Overall, the dynamics of vertical vergence resemble the dynamics of horizontal vergence and cyclovergence.

Binocular correlation detection with oriented dynamic random-line stereograms.

Stereopsis relies principally on the extraction of horizontal retinal disparities. As such, we assume that the vertical contours (i.e., horizontal contrast energy) are of principle import for stereopsis. Yet there are theoretical reasons for believing that horizontal contours should be involved in binocular matching (if not stereopsis proper) as well. First, they would facilitate the computation of vertical disparities, which are necessary for the control of disjunctive eye movements and perhaps the computation of absolute depth. Second, the process of binocular matching is a two-dimensional one; its solution requires information along both principle orientations. In this study, we have measured the efficacy with which horizontal or vertical contours can be binocularly matched by measuring thresholds for the detection of interocular correlation for oriented dynamic random-line stereograms. We find that the slopes of the psychometric functions are almost a factor of two steeper when matching vertical contours, indicating a narrower noise distribution along the decision axis associated with these stimuli.

Hyperglobal Interactions in Perception of Stereoscopic Transparency

A method has been developed (1) to measure limitations of human observers in perception of visual scenes containing a transparent surface, and (2) to study computational stages underlying stereoscopic transparency perception. Observers viewed random-dot stereograms of overlapping plane and cylindrical surfaces and had to distinguish between two orientations of the cylindrical surface under conditions of strict depth fixation control. Dot density of the transparent plane was increased at various intersurface depth separations until identification of cylinder orientation became random (limiting density). Limiting density dramatically decreased as depth separation between the surfaces grew, and this basic relationship could not be accounted for either by higher severity of matching with larger dot densities or by the ability of a denser surface to pull vergence to its depth. The basic relationship persisted with opposite contrast polarities of plane and cylinder dots, which suggests that competitive interactions between the surfaces occur as a separate process from binocular matching at each surface [Harris and Parker, 1995 Nature (London)374 808 – 811]. Thus, successive computational stages of matching and intersurface interaction characterising stereoscopic hyperglobality may be distinguished. We also found a facilitating role of intersurface brightness difference and a near — far asymmetry of the basic limiting density versus depth separation relationship with reversed surface positioning. We suggest a scheme of cortical inhibitory connections that agrees with the concept of disparity gradient limit and predicts performance similar to that reported. In the scheme, each activated unit produces a suppression zone in the disparity projection field so that lateral extension of the inhibition grows with binocular disparity.

Multiple matching of features in simple stereograms

Simple stereograms are used to show that the binocular matching of closely spaced vertical lines can be changed by horizontally connecting some of the vertical lines. The matching that is seen can be gradually changed by incremental modifications to simple visual quantities like the luminance contrast of a local region, or the density of the connecting horizontal lines. For some values of these visual quantities the depth seen is unstable which suggests that the mechanisms responsible for resolving matching ambiguities might be intrinsically dynamic. These changes in binocular matching can also be used to show a hysteresis-like effect in stereopsis. Many of the constraints like ordering, cohesivity, and uniqueness often used in algorithms for resolving ambiguities are perhaps too limiting to allow either such dynamics or the processing of transparent or partially occluded surfaces.

Binocular correspondence in stereoscopic vision.

For binocular stereoscopic vision to be possible, the visual nervous system needs to perform two tasks with the information available from the left and right eyes. First, features visible from the left eye must be paired up with the corresponding feature as seen from the right eye's vantage point. Second, the geometric information from the matched binocular features must be transformed into an estimate of binocular disparity. In this sense, the paper by Barlow, Blakemore and Pettigrew1 represents the first complete proposal for the neural mechanisms underlying binocular correspondence and the per­ception of stereoscopic depth. They proposed that the feature selectivity of visually-responsive neurons should have a central role in sorting out which visual features in the left eye should match with those in the right eye and that the same group of neurons should provide signals from which binocular disparity could be extracted to signal depth. In short, for the cat's visual system, the neurons in cortical area VI were proposed to be involved in both binocular matching and the recovery of stereoscopic depth. Neurons in VI have a number of obvious monocular feature selectivities that can be exploited for binocular matching, namely the local orientation, spatial frequency and spatial phase of regions of the image's luminance (black/white) contrast. The beha­vioural significance of these features is also con­firmed by numerous psychophysical studies of the stereoscopic capabilities of human vision. Other feature selectivities in VI, such as colour, may have

An upper limit to the binocular combination of stimuli

It is important to know the spatial extent over which the binocular visual system searches for “matches” or image correspondence. Most models of stereopsis define fixed neighbourhoods in one monocular image in which a search is conducted for a match to some element in the other image. We were unable to experimentally determine fixed values for the extent of these neighbourhoods. We were, however, able to derive a simple rule that predicts performance on a binocular matching task over a large spatio-temporal region, and from which we can calculate the efficiency of our observers. We conclude that the human visual system does not use neighbourhoods of a single, fixed extent to perform binocular matching in a single region of the visual field, but uses more flexible scheme that allows it to function well under a broad range of stimulus conditions.