Stereoscopic Surface Research Articles

Adaptive infrared patterns for microscopic surface reconstructions.

Multi-zoom microscopic surface reconstructions of operating sites, especially in ENT surgeries, would allow multimodal image fusion for determining the amount of resected tissue, for recognizing critical structures, and novel tools for intraoperative quality assurance. State-of-the-art three-dimensional model creation of the surgical scene is challenged by the surgical environment, illumination, and the homogeneous structures of skin, muscle, bones, etc., that lack invariant features for stereo reconstruction. An adaptive near-infrared pattern projector illuminates the surgical scene with optimized patterns to yield accurate dense multi-zoom stereoscopic surface reconstructions. The approach does not impact the clinical workflow. The new method is compared to state-of-the-art approaches and is validated by determining its reconstruction errors relative to a high-resolution 3D-reconstruction of CT data. 200 surface reconstructions were generated for 5 zoom levels with 10 reconstructions for each object illumination method (standard operating room light, microscope light, random pattern and adaptive NIR pattern). For the adaptive pattern, the surface reconstruction errors ranged from 0.5 to 0.7 mm, as compared to 1-1.9 mm for the other approaches. The local reconstruction differences are visualized in heat maps. Adaptive near-infrared (NIR) pattern projection in microscopic surgery allows dense and accurate microscopic surface reconstructions for variable zoom levels of small and homogeneous surfaces. This could potentially aid in microscopic interventions at the lateral skull base and potentially open up new possibilities for combining quantitative intraoperative surface reconstructions with preoperative radiologic imagery.

Space–time statistics of 2D soliton gas in shallow water studied by stereoscopic surface mapping

Space–time statistics of 2D soliton gas in shallow water studied by stereoscopic surface mapping

Stereoscopic slant contrast revisited.

The perceived slant of a stereoscopic surface is altered by the presence of a surrounding surface, a phenomenon termed stereo slant contrast. Previous studies have shown that a slanted surround causes a fronto-parallel surface to appear slanted in the opposite direction, an instance of "bidirectional" contrast. A few studies have examined slant contrast using slanted as opposed to fronto-parallel test surfaces, and these also have shown slant contrast. Here, we use a matching method to examine slant contrast over a wide range of combinations of surround and test slants, one aim being to determine whether stereo slant contrast transfers across opposite directions of test and surround slant. We also examine the effect of the test on the perceived slant of the surround. Test slant contrast was found to be bidirectional in virtually all test-surround combinations and transferred across opposite test and surround slants, with little or no decline in magnitude as the test-surround slant difference approached the limit. There was a weak bidirectional effect of the test slant on the perceived slant of the surround. We consider how our results might be explained by four mechanisms: (a) normalization of stereo slant to vertical; (b) divisive normalization of stereo slant channels in a manner analogous to the tilt illusion; (c) interactions between center and surround disparity-gradient detectors; and (d) uncertainty in slant estimation. We conclude that the third of these (interactions between center and surround disparity-gradient detectors) is the most likely cause of stereo slant contrast.

Depth perception of stereoscopic transparent stimuli with frame manipulation

Depth perception is crucial in human vision, allowing us to move and interact with our 3-D surroundings. We used a stereoscopic transparent stimulus comprising parallel overlapping transparent stereoscopic surfaces (POTS) to understand depth perception better. The study focused on exploring the effect of a surrounding frame on the perceived depth of a POTS configuration. The research was based on a proposed idea that explains an "off-frame" effect: a frame at a different depth from a 2-D photograph depicting a 3-D scene increases its apparent depth qualitatively. The idea assumes that processing the disparity between a frame and a photo reduces the reliability of the photograph's flatness cues and increases depth magnitude in depth cue integration. We examined whether the idea can be applied to a 3-D POTS with the flatness cue as the constant accommodation. Through three experiments, the study showed that frames impact the perceived depth magnitude of a POTS configuration. More specifically, the depth magnitude increases as the frame's disparity concerning the monitor plane increases and decreases as the frame's size increases. We discussed the results in the context of depth cue combination.

Spatially resolved stereoscopic surface profiling by using a feature-selective segmentation and merging technique

We present a feature-selective segmentation and merging technique to achieve spatially resolved surface profiles of the parts by 3D stereoscopy and strobo-stereoscopy. A pair of vision cameras capture images of the parts at different angles, and 3D stereoscopic images can be reconstructed. Conventional filtering processes of the 3D images involve data loss and lower the spatial resolution of the image. In this study, the 3D reconstructed image was spatially resolved by automatically recognizing and segmenting the features on the raw images, locally and adaptively applying super-resolution algorithm to the segmented images based on the classified features, and then merging those filtered segments. Here, the features are transformed into masks that selectively separate the features and background images for segmentation. The experimental results were compared with those of conventional filtering methods by using Gaussian filters and bandpass filters in terms of spatial frequency and profile accuracy. As a result, the selective feature segmentation technique was capable of spatially resolved 3D stereoscopic imaging while preserving imaging features.

Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study

BackgroundThe laparoscopic approach to liver resection may reduce morbidity and hospital stay. However, uptake has been slow due to concerns about patient safety and oncological radicality. Image guidance systems may improve patient safety by enabling 3D visualisation of critical intra- and extrahepatic structures. Current systems suffer from non-intuitive visualisation and a complicated setup process. A novel image guidance system (SmartLiver), offering augmented reality visualisation and semi-automatic registration has been developed to address these issues. A clinical feasibility study evaluated the performance and usability of SmartLiver with either manual or semi-automatic registration.MethodsIntraoperative image guidance data were recorded and analysed in patients undergoing laparoscopic liver resection or cancer staging. Stereoscopic surface reconstruction and iterative closest point matching facilitated semi-automatic registration. The primary endpoint was defined as successful registration as determined by the operating surgeon. Secondary endpoints were system usability as assessed by a surgeon questionnaire and comparison of manual vs. semi-automatic registration accuracy. Since SmartLiver is still in development no attempt was made to evaluate its impact on perioperative outcomes.ResultsThe primary endpoint was achieved in 16 out of 18 patients. Initially semi-automatic registration failed because the IGS could not distinguish the liver surface from surrounding structures. Implementation of a deep learning algorithm enabled the IGS to overcome this issue and facilitate semi-automatic registration. Mean registration accuracy was 10.9 ± 4.2 mm (manual) vs. 13.9 ± 4.4 mm (semi-automatic) (Mean difference − 3 mm; p = 0.158). Surgeon feedback was positive about IGS handling and improved intraoperative orientation but also highlighted the need for a simpler setup process and better integration with laparoscopic ultrasound.ConclusionThe technical feasibility of using SmartLiver intraoperatively has been demonstrated. With further improvements semi-automatic registration may enhance user friendliness and workflow of SmartLiver. Manual and semi-automatic registration accuracy were comparable but evaluation on a larger patient cohort is required to confirm these findings.

Generalized Representation of Stereoscopic Surface Shape and Orientation in the Human Visual Cortex.

The brain’s ability to extract three-dimensional (3D) shape and orientation information from viewed objects is vital in daily life. Stereoscopic 3D surface perception relies on binocular disparity. Neurons selective to binocular disparity are widely distributed among visual areas, but the manner in these areas are involved in stereoscopic 3D surface representation is unclear. To address this, participants were instructed to observe random dot stereograms (RDS) depicting convex and concave curved surfaces and the blood oxygenation level-dependent (BOLD) signal of visual cortices was recorded. Two surface types were: (i) horizontally positioned surfaces defined by shear disparity; and (ii) vertically positioned surfaces defined by compression disparity. The surfaces were presented at different depth positions per trial. Functional magnetic resonance imaging (fMRI) data were classified from early visual areas to higher visual areas. We determined whether cortical areas were selective to shape and orientation by assessing same-type stimuli classification accuracies based on multi-voxel activity patterns per area. To identify whether some areas were related to a more generalized sign of curvature or orientation representation, transfer classification was used by training classifiers on one dataset type and testing classifiers on another type. Same-type stimuli classification results showed that most selected visual areas were selective to shape and all were selective to the orientation of disparity-defined 3D surfaces. Transfer classification results showed that in the dorsal visual area V3A, classification accuracies for the discriminate sign of surface curvature were higher than the baseline of statistical significance for all types of classifications, demonstrating that V3A is related to generalized shape representation. Classification accuracies for discriminating horizontal–vertical surfaces in higher dorsal areas V3A and V7 and ventral area lateral occipital complex (LOC) as well as in some areas of intraparietal sulcus (IPS) were higher than the baseline of statistical significance, indicating their relation to the generalized representation of 3D surface orientation.

Computer simulation and system realization of jacquard weft-knitted fabric

Jacquard weft-knitted fabric is a type of multilayer knitting fabric which has a stereoscopic surface with complex patterns. A computer simulation system was established, based on a virtual fabric Unity3D platform, to minimize the time required for computer simulation and to enhance the simulation results. The three-dimensional fabric data were unified and coordinate data were obtained to achieve the establishment. Three factors of simulation algorithm for jacquard weft-knitted fabric were used to establish and control the effect of a fabric model. By integrating three-dimensional model data, two-dimensional spatial data, and gray model, virtual simulation based on three-dimensional engine Unity3D platform was realized. Based on the real-fabric data, the final fabric patterns were predicted from macroscopic angle instead of two-dimensional and microscopic simulations. The system is proved to be more effective for the prediction and simulation of jacquard weft-knitted fabric and other fabrics having similar stereoscopic surfaces.

Generalized representation of stereoscopic surface in V3A

Generalized representation of stereoscopic surface in V3A

Early dynamics of stereoscopic surface slant perception.

Surface orientation is an important visual primitive that can be estimated from monocular or binocular (stereoscopic) signals. Changes in motor planning occur within about 200 ms after either type of signal is perturbed, but the time it takes for apparent (perceived) slant to develop from stereoscopic cues is not known. Apparent slant sometimes develops very slowly (Gillam, Chambers, & Russo, 1988; van Ee & Erkelens, 1996). However, these long durations could reflect the time it takes for the visual system to resolve conflicts between slant cues that inevitably specify different slants in laboratory displays (Allison & Howard, 2000). We used a speed–accuracy tradeoff analysis to measure the time it takes to discriminate slant, allowing us to report psychometric functions as a function of response time. Observers reported which side of a slanted surface was farther, with a temporal deadline for responding that varied block-to-block. Stereoscopic slant discrimination rose above chance starting at 200 ms after stimulus onset. Unexpectedly, observers discriminated slant from binocular disparity faster than texture, and for stereoscopic whole-field stimuli faster than stereoscopic slant contrast stimuli. However, performance after the initial deviation from chance increased more rapidly for slant-contrast stimuli than whole-field stimuli. Discrimination latencies were similar for slants about the horizontal and vertical axes, but performance increased faster for slants about the vertical axis. Finally, slant from vertical disparity was somewhat slower than slant from horizontal disparity, which may reflect cue conflict. These results demonstrate, in contradiction with the previous literature, that the perception of slant from disparity happens very quickly—in fact, more quickly than the perception of slant from texture—and in comparable time to the simple perception of brightness from luminance.

On the combination of illusory and luminance-defined stereoscopic surfaces

On the combination of illusory and luminance-defined stereoscopic surfaces

Distortions in perceived depth magnitude for stereoscopic surfaces

Distortions in perceived depth magnitude for stereoscopic surfaces

Stereoscopic surface interpolation from illusory contours

Stereoscopic Kanizsa figures are an example of stereoscopic interpolation of an illusory surface. In such stimuli, luminance-defined disparity signals exist only along the edges of inducing elements, but observers reliably perceive a coherent surface that extends across the central region in depth. The aim of this series of experiments was to understand the nature of the disparity signal that underlies the perception of illusory stereoscopic surfaces. I systematically assessed the accuracy and precision of suprathreshold depth percepts using a collection of Kanizsa figures with a wide range of 2D and 3D properties. For comparison, I assessed similar perceptually equated figures with luminance-defined surfaces, with and without inducing elements. A cue combination analysis revealed that observers rely on ordinal depth cues in conjunction with stereopsis when making depth judgements. Thus, 2D properties (e.g. occlusion features and luminance relationships) contribute rich information about 3D surface structure by influencing perceived depth from binocular disparity.

Generic epipolar resampling method for perspective frame camera and linear push-broom sensor

Epipolar rectification aims at resampling stereoscopic images so that conjugate points are located along the same horizontal x-axis. For most stereoscopic digital surface model production algorithms, this rectification is a necessary preprocessing task because it allows reducing the 2D matching to a 1D matching problem. Most epipolar rectification algorithms exist in literature and are mainly for a given type of sensor (frame camera or push-broom sensor). In this paper, we present a generic method for epipolar rectification. This means that the method is applicable for both frame and push-broom cameras. The only difference lies in the projection function used to generate epipolar curves.

Gradients of relative disparity underlie the perceived slant of stereoscopic surfaces.

Perceived stereoscopic slant around a vertical axis is strongly underestimated for isolated surfaces, suggesting that neither uniocular image compression nor linear gradients of absolute disparity are very effective cues. However, slant increases to a level close to geometric prediction if gradients of relative disparity are introduced, for example by placing flanking frontal-parallel surfaces at the horizontal boundaries of the slanted surface. Here we examine the mechanisms underlying this slant enhancement by manipulating properties of the slanted surface or the flanking surfaces. Perceived slant was measured using a probe bias method. In Experiment 1, an outlined surface and a randomly textured surface showed similar slant underestimation when presented in isolation, but the enhancement in slant produced by flankers was significantly greater for the textured surface. In Experiment 2, we degraded the relative disparity gradient by (a) reducing overall texture density, (b) reducing flanker width, or (c) adding disparity noise to the flankers. Density had no effect while adding noise to the flankers, or reducing their width significantly decreased perceived slant of the central surface. These results support the view that the enhancement of slant produced by adding flanking surfaces is attributable to the presence of a relative disparity gradient and that the flanker effect can spread to regions of the surface not directly above or below the gradient.

Evidence of Stereoscopic Surface Disambiguation in the Responses of V1 Neurons.

For the important task of binocular depth perception from complex natural-image stimuli, the neurophysiological basis for disambiguating multiple matches between the eyes across similar features has remained a long-standing problem. Recurrent interactions among binocular disparity-tuned neurons in the primary visual cortex (V1) could play a role in stereoscopic computations by altering responses to favor the most likely depth interpretation for a given image pair. Psychophysical research has shown that binocular disparity stimuli displayed in 1 region of the visual field can be extrapolated into neighboring regions that contain ambiguous depth information. We tested whether neurons in macaque V1 interact in a similar manner and found that unambiguous binocular disparity stimuli displayed in the surrounding visual fields of disparity-selective V1 neurons indeed modified their responses when either bistable stereoscopic or uniform featureless stimuli were presented within their receptive field centers. The delayed timing of the response behavior compared with the timing of classical surround suppression and multiple control experiments suggests that these modulations are carried out by slower disparity-specific recurrent connections among V1 neurons. These results provide explicit evidence that the spatial interactions that are predicted by cooperative algorithms play an important role in solving the stereo correspondence problem.

Binocular contributions to linear vertical vection.

Compelling illusions of self-motion, known as vection, can be produced in a stationary observer by visual stimulation alone. The role of binocular vision and stereopsis in these illusions was explored in a series of three experiments. Previous research had provided evidence of stereoscopic enhancements for linear vection in depth (e.g., Palmisano, 1996, 2002). Here we examined for the first time the effects of binocular vision and stereopsis on linear vertical vection. Vertical vection was induced by the upward or downward translation of large stereoscopic surfaces. These surfaces were horizontally oriented depth corrugations produced by disparity modulation of patterns of persistent or short lifetime dot elements. We found that binocular viewing of such surfaces significantly increased the magnitudes and decreased the onset delays of vertical vection. Experiments utilizing short lifetime dot stereograms demonstrated that these particular binocular enhancements of vection were due to the motion of stereoscopically defined features.

Magnitude of perceived depth of multiple stereo transparent surfaces.

According to the geometric relational expression of binocular stereopsis, for a given viewing distance the magnitude of the perceived depth of objects would be the same, as long as the disparity magnitudes were the same. However, we found that this is not necessarily the case for random-dot stereograms that depict parallel, overlapping, transparent stereoscopic surfaces (POTS). The data from five experiments indicated that (1) the magnitude of perceived depth between the two outer surfaces of a three- or a four-POTS configuration can be smaller than that for an identical pair of stereo surfaces of a two-POTS configuration for the range of disparities that we used (5.2-19.4 arcmin); (2) this phenomenon can be observed irrespective of the total dot density of a POTS configuration, at least for the range that we used (1.1-3.3 dots/deg(2)); and (3) the magnitude of perceived depth between the two outer surfaces of a POTS configuration can be reduced as the total number of stereo surfaces is increased, up to four surfaces. We explained these results in terms of a higher-order process or processes, with an output representing perceived depth magnitude, which is weakened when the number of its surfaces is increased.

The role of monocular regions in the perception of stereoscopic surfaces

Abstract presented at the 36th European Conference on Visual Perception, 25-29 August 2013, Bremen, Germany

Monocular and binocular edges enhance the perception of stereoscopic slant.

Gradients of absolute binocular disparity across a slanted surface are often considered the basis for stereoscopic slant perception. However, perceived stereo slant around a vertical axis is usually slow and significantly under-estimated for isolated surfaces. Perceived slant is enhanced when surrounding surfaces provide a relative disparity gradient or depth step at the edges of the slanted surface, and also in the presence of monocular occlusion regions (sidebands). Here we investigate how different kinds of depth information at surface edges enhance stereo slant about a vertical axis. In Experiment 1, perceived slant decreased with increasing surface width, suggesting that the relative disparity between the left and right edges was used to judge slant. Adding monocular sidebands increased perceived slant for all surface widths. In Experiment 2, observers matched the slant of surfaces that were isolated or had a context of either monocular or binocular sidebands in the frontal plane. Both types of sidebands significantly increased perceived slant, but the effect was greater with binocular sidebands. These results were replicated in a second paradigm in which observers matched the depth of two probe dots positioned in front of slanted surfaces (Experiment 3). A large bias occurred for the surface without sidebands, yet this bias was reduced when monocular sidebands were present, and was nearly eliminated with binocular sidebands. Our results provide evidence for the importance of edges in stereo slant perception, and show that depth from monocular occlusion geometry and binocular disparity may interact to resolve complex 3D scenes.