Stereo Fusion Research Articles

Stereo Superpixel Segmentation via Decoupled Dynamic Spatial-Embedding Fusion Network

Stereo superpixel segmentation aims at grouping the discretizing pixels into perceptual regions through left and right views more collaboratively and efficiently. Existing superpixel segmentation algorithms mostly utilize color and spatial features as input, which may impose strong constraints on spatial information while utilizing the disparity information in terms of stereo image pairs. To alleviate this issue, we propose a stereo superpixel segmentation method with a decoupling mechanism of spatial information in this work. To decouple stereo disparity information and spatial information, the spatial information is temporarily removed before fusing the features of stereo image pairs, and a decoupled stereo fusion module (DSFM) is designed to handle the stereo features alignment as well as occlusion problems. Moreover, since the spatial information is vital to superpixel segmentation, we further design a dynamic spatiality embedding module (DSEM) to re-add spatial information, and the weights of spatial information will be adaptively adjusted through the dynamic fusion (DF) mechanism in DSEM for achieving a finer segmentation. Comprehensive experimental results demonstrate that our method can achieve the state-of-the-art performance on the KITTI2015 and Cityscapes datasets, and also verify the efficiency when applied in salient object detection on NJU2K dataset. The source code will be available publicly after paper is accepted.

Cross-View Attention Interaction Fusion Algorithm for Stereo Super-Resolution

In the process of stereo super-resolution reconstruction, in addition to the richness of the extracted feature information directly affecting the texture details of the reconstructed image, the texture details of the corresponding pixels between stereo image pairs also have an important impact on the reconstruction accuracy in the process of network learning. Therefore, aiming at the information interaction and stereo consistency of stereo image pairs, a cross-view attention interaction fusion stereo super-resolution algorithm is proposed. Firstly, based on parallax attention mechanism and triple attention mechanism, an attention stereo fusion module is constructed. The attention stereo fusion module is inserted between different levels of two single image super-resolution network branches, and the attention weight is calculated through the cross dimensional interaction of the three branches. It makes full use of the ability of single image super-resolution network to extract single view information and further maintaining the stereo consistency between stereo image pairs. Then, an enhanced cross-view interaction strategy including three fusion methods is proposed. Specifically, the vertical sparse fusion method is used to integrate the interior view information of different levels in the two single image super-resolution sub branches, the horizontal dense fusion method is used to connect the adjacent attention stereo fusion modules and the constraint between stereo image consistency is further strengthened in combination with the feature fusion method. Finally, the experimental results on Flickr 1024, Middlebury and KITTI benchmark datasets show that the proposed algorithm is superior to the existing stereo image super-resolution methods in quantitative measurement and qualitative visual quality while maintaining the stereo consistency of image pairs.

3D Object Detection for Self-Driving Cars Using Video and LiDAR: An Ablation Study.

Methods based on 64-beam LiDAR can provide very precise 3D object detection. However, highly accurate LiDAR sensors are extremely costly: a 64-beam model can cost approximately USD 75,000. We previously proposed SLS-Fusion (sparse LiDAR and stereo fusion) to fuse low-cost four-beam LiDAR with stereo cameras that outperform most advanced stereo-LiDAR fusion methods. In this paper, and according to the number of LiDAR beams used, we analyzed how the stereo and LiDAR sensors contributed to the performance of the SLS-Fusion model for 3D object detection. Data coming from the stereo camera play a significant role in the fusion model. However, it is necessary to quantify this contribution and identify the variations in such a contribution with respect to the number of LiDAR beams used inside the model. Thus, to evaluate the roles of the parts of the SLS-Fusion network that represent LiDAR and stereo camera architectures, we propose dividing the model into two independent decoder networks. The results of this study show that-starting from four beams-increasing the number of LiDAR beams has no significant impact on the SLS-Fusion performance. The presented results can guide the design decisions by practitioners.

3D Object Detection with SLS-Fusion Network in Foggy Weather Conditions.

The role of sensors such as cameras or LiDAR (Light Detection and Ranging) is crucial for the environmental awareness of self-driving cars. However, the data collected from these sensors are subject to distortions in extreme weather conditions such as fog, rain, and snow. This issue could lead to many safety problems while operating a self-driving vehicle. The purpose of this study is to analyze the effects of fog on the detection of objects in driving scenes and then to propose methods for improvement. Collecting and processing data in adverse weather conditions is often more difficult than data in good weather conditions. Hence, a synthetic dataset that can simulate bad weather conditions is a good choice to validate a method, as it is simpler and more economical, before working with a real dataset. In this paper, we apply fog synthesis on the public KITTI dataset to generate the Multifog KITTI dataset for both images and point clouds. In terms of processing tasks, we test our previous 3D object detector based on LiDAR and camera, named the Spare LiDAR Stereo Fusion Network (SLS-Fusion), to see how it is affected by foggy weather conditions. We propose to train using both the original dataset and the augmented dataset to improve performance in foggy weather conditions while keeping good performance under normal conditions. We conducted experiments on the KITTI and the proposed Multifog KITTI datasets which show that, before any improvement, performance is reduced by 42.67% in 3D object detection for Moderate objects in foggy weather conditions. By using a specific strategy of training, the results significantly improved by 26.72% and keep performing quite well on the original dataset with a drop only of 8.23%. In summary, fog often causes the failure of 3D detection on driving scenes. By additional training with the augmented dataset, we significantly improve the performance of the proposed 3D object detection algorithm for self-driving cars in foggy weather conditions.

Robustness of ToF and stereo fusion for high‐accuracy depth map

A depth map can be used in many applications such as robotic navigation, driverless, video production and 3D reconstruction. Both passive stereo and time‐of‐flight (ToF) cameras can provide the depth map for the captured real scenes, but they both have innate limitations. Since ToF cameras and passive stereo are intrinsically complementary for certain scenes, it is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. As a result, this study proposes an approach to integrate ToF cameras and passive stereo to obtain high‐accuracy depth maps. The main contributions are: the first step is to design an energy cost function to utilise the depth map from ToF cameras to guide the stereo matching of passive stereo and the second step is to design their weight function for depth maps pixel‐level fusion. The experiments show that the proposed approach achieves the improved results with high accuracy and robustness.

Three-dimensional reconstruction based on multi-view photometric stereo fusion technology in movies special-effect

3D movies have become very popular in recent years. But there are vertical disparities between left and right views of 3D frames due to the lack of accuracy of the mechanical alignment during the shooting. In order to improve accuracy of reconstruction, a three-dimensional reconstruction technology based on multi-view photometric stereo fusion algorithm in movies special-effect production is present in this paper. The original normal is firstly replaced with the surface normal in the average normal, and the reconstructed normal is optimized so as to reduce the deviation of the original surface normal. And then, a reference-plane-based approach is applied to estimate the principle optical axis of each light source as well as its principle radiant energy. For each surface point on the target, the direction and intensity of its incident light ray can be precisely determined by the calibration parameters and the quasi-point light model. Finally, 3D reconstruction of the surface with a quasi-point light source is also implemented in two steps. By estimating the mean value of depth in the iterative process, the surface depth is projected into the physical coordinates. Qualitative and quantitative experimental results show that higher accuracy surface normal as well as better 3D reconstruction quality can be obtained by the proposed approach in comparison with conventional reconstruction methods.

Robustness of ToF and Stereo Fusion for High Accuracy Depth Maps

Depth map can be used in many applications such as robotic navigation, driverless, video production and 3D reconstruction. Currently, both active depth cameras and passive stereo vision systems are main technical means to obtain depth information, but each of two systems alone has its own limitations. In this paper, Time-of-Flight (ToF) and stereo fusion framework is proposed to solve limitations of these two systems. The scheme in this paper contains the prior fusion stage and the post fusion stage. Using depth information design energy function from ToF cameras to boost stereo matching of passive stereo in the prior fusion stage. During the post fusion stage, weighting function is designed according to depth map of stereo matching and credibility map of ToF depth map, and then adaptive weighted depth fusion is performed. Experimental results clearly show that our method can obtain high-precision and high-resolution depth maps with better robustness than other fusion approaches.

Perspective Preserving Solution for Quasi-Orthoscopic Video See-Through HMDs

In non-orthoscopic video see-through (VST) head-mounted displays (HMDs), depth perception through stereopsis is adversely affected by sources of spatial perception errors. Solutions for parallax-free and orthoscopic VST HMDs were considered to ensure proper space perception but at expenses of an increased bulkiness and weight. In this work, we present a hybrid video-optical see-through HMD the geometry of which explicitly violates the rigorous conditions of orthostereoscopy. For properly recovering natural stereo fusion of the scene within the personal space in a region around a predefined distance from the observer, we partially resolve the eye-camera parallax by warping the camera images through a perspective preserving homography that accounts for the geometry of the VST HMD and refers to such distance. For validating our solution; we conducted objective and subjective tests. The goal of the tests was to assess the efficacy of our solution in recovering natural depth perception in the space around said reference distance. The results obtained showed that the quasi-orthoscopic setting of the HMD; together with the perspective preserving image warping; allow the recovering of a correct perception of the relative depths. The perceived distortion of space around the reference plane proved to be not as severe as predicted by the mathematical models.

Multi-Scale 7DOF View Adjustment.

Multi-scale virtual environments contain geometric details ranging over several orders of magnitude and typically employ out-of-core rendering techniques. When displayed in virtual reality systems this entails using a 7 degree-of-freedom (DOF) view model where view scale is a separate 7th DOF in addition to 6DOF view pose. Dynamic adjustment of this and other view parameters become very important to usability. In this paper, we evaluate how two adjustment techniques interact with uni- and bi-manual 7DOF navigation in DesktopVR and a CAVE. The travel task has two stages, an initial targeted zoom and a detailed geometric inspection. The results show benefits of the auto-adjustments on completion time and stereo fusion issues, but only in certain circumstances. Peculiar view configuration examples show the difficulty of creating robust adjustment rules.

Stereo fusion: Combining refractive and binocular disparity

The performance of depth reconstruction in binocular stereo relies on how adequate the predefined baseline for a target scene is. Wide-baseline stereo is capable of discriminating depth better than the narrow-baseline stereo, but it often suffers from spatial artifacts. Narrow-baseline stereo can provide a more elaborate depth map with fewer artifacts, while its depth resolution tends to be biased or coarse due to the short disparity. In this paper, we propose a novel optical design of heterogeneous stereo fusion on a binocular imaging system with a refractive medium, where the binocular stereo part operates as wide-baseline stereo, and the refractive stereo module works as narrow-baseline stereo. We then introduce a stereo fusion workflow that combines the refractive and binocular stereo algorithms to estimate fine depth information through this fusion design. In addition, we propose an efficient calibration method for refractive stereo. The quantitative and qualitative results validate the performance of our stereo fusion system in measuring depth in comparison with homogeneous stereo approaches.

A New High Resolution Depth Map Estimation System Using Stereo Vision and Kinect Depth Sensing

Depth map estimation is an active and long standing problem in image/video processing and computer vision. Conventional depth estimation algorithms which rely on stereo/multi-view vision or depth sensing devices alone are limited by complicated scenes or imperfections of the depth sensing devices. On the other hand, the depth maps obtained from the stereo/multi-view vision and depth sensing devices are de facto complementary to each other. This motivates us to develop in this paper a new system for high resolution and high quality depth estimation by joint fusion of stereo and Kinect data. We modeled the observations using Markov random field (MRF) and formulated the fusion problem as a maximum a posteriori probability (MAP) estimation problem. The reliability and the probability density functions for describing the observations from the two devices are also derived. The MAP problem is solved using a multiscale belief propagation (BP) algorithm. To suppress possible estimation noise, the depth map estimated is further refined by color image guided depth matting and a 2D polynomial regression (LPR)-based filtering. Experimental results and numerical comparisons show that our system can provide high quality and high resolution depth maps, thanks to the complementary strengths of both stereo vision and Kinect depth sensors.

Benefits of Matching Accommodative Demands to Vergence Demands in a Binocular Head-Mounted Display: A Study on Stereo Fusion Times

Current head-mounted displays (HMDs) provide only a fixed lens focus. Viewers have to decouple their accommodation and vergence responses when viewing stereoscopic images presented on an HMD. This study investigates the time taken to fuse a pair of stereoscopic images displayed on an HMD when the accommodative demand is matched to the vergence demand. Four testing conditions exhausting the factorial combinations of accommodative demands (2.5 D and 0.5 D) and vergence demands (2.5 MA and 0.5 MA) were investigated. The results indicate that viewers take a significantly shorter amount of time to fuse a pair of stereoscopic images (i.e., fusion time) when the accommodative demand and the stereoscopic depth cues match. Further analysis suggests that an unnatural demand for the eyes to verge toward stereoscopic images whose stereo depth is farther than the accommodative demand is associated with significantly longer fusion time. This study evaluates the potential benefits of using a dynamically adjustable lens focus in future designs of HMDs.

Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains

We propose a convex formulation for silhouette and stereo fusion in 3D reconstruction from multiple images. The key idea is to show that the reconstruction problem can be cast as one of minimizing a convex functional, where the exact silhouette consistency is imposed as convex constraints that restrict the domain of feasible functions. As a consequence, we can retain the original stereo-weighted surface area as a cost functional without heuristic modifications of this energy by balloon terms or other strategies, yet still obtain meaningful (non-empty) reconstructions which are guaranteed to be silhouette-consistent. We prove that the proposed convex relaxation approach provides solutions that lie within a bound of the optimal solution. Compared to existing alternatives, the proposed method does not depend on initialization and leads to a simpler and more robust numerical scheme for imposing silhouette consistency obtained by projection onto convex sets. We show that this projection can be solved exactly using an efficient algorithm. We propose a parallel implementation of the resulting convex optimization problem on a graphics card. Given a photo-consistency map and a set of image silhouettes, we are able to compute highly accurate and silhouette-consistent reconstructions for challenging real-world data sets. In particular, experimental results demonstrate that the proposed silhouette constraints help to preserve fine-scale details of the reconstructed shape. Computation times depend on the resolution of the input imagery and vary between a few seconds and a couple of minutes for all experiments in this paper.

Volumetric stereo and silhouette fusion for image-based modeling

This paper presents a volumetric stereo and silhouette fusion algorithm for acquiring high quality models from multiple calibrated photographs. Our method is based on computing and merging depth ma...

Probabilistic fusion of stereo with color and contrast for bilayer segmentation

This paper describes models and algorithms for the real-time segmentation of foreground from background layers in stereo video sequences. Automatic separation of layers from color/contrast or from stereo alone is known to be error-prone. Here, color, contrast, and stereo matching information are fused to infer layers accurately and efficiently. The first algorithm, Layered Dynamic Programming (LDP), solves stereo in an extended six-state space that represents both foreground/background layers and occluded regions. The stereo-match likelihood is then fused with a contrast-sensitive color model that is learned on-the-fly and stereo disparities are obtained by dynamic programming. The second algorithm, Layered Graph Cut (LGC), does not directly solve stereo. Instead, the stereo match likelihood is marginalized over disparities to evaluate foreground and background hypotheses and then fused with a contrast-sensitive color model like the one used in LDP. Segmentation is solved efficiently by ternary graph cut. Both algorithms are evaluated with respect to ground truth data and found to have similar performance, substantially better than either stereo or color/ contrast alone. However, their characteristics with respect to computational efficiency are rather different. The algorithms are demonstrated in the application of background substitution and shown to give good quality composite video output.

Stereovision imaging on submersible platforms for 3-D mapping of benthic habitats and sea-floor structures

We investigate the deployment of a submersible platform with stereovision imaging capability for three-dimensional (3D) mapping of benthic habitats and other sea-floor structures over local areas. A complete framework is studied, comprising: 1) suitable trajectories to be executed for data collection; 2) data processing for positioning and trajectory followed by online frame-to-frame and frame-to-mosaic registration of images, as well as recursive global realignment of positions along the path; and 3) 3D mapping by the fusion of various visual cues, including motion and stereo within a Kalman filter. The computational requirements of the system are evaluated, formalizing how processing may be achieved in real time. The proposed scenario is simulated for testing with known ground truth to assess the system performance, to quantify various errors, and to identify how performance may be improved. Experiments with underwater images are also presented to verify the performance of various components and the overall scheme.

An obstacle detection method by fusion of radar and motion stereo

In order to avoid collision with an object that blocks the course of a vehicle, measuring the distance to it and detecting positions of its side boundaries, are necessary. In the paper, an object detection method achieved by the fusion of millimeter-wave radar and a single video camera is proposed. We consider the method as the least expensive solution because at least one camera is necessary for lane marking detection. In the method, the distance is measured by the radar, and the boundaries are found from an image sequence, based on a motion stereo technique with the help of the distance measured by the radar. Since the method does not depend on the appearance of objects, it is capable of detecting not only an automobile but also other objects. Object detection by the method was confirmed through an experiment. In the experiment, both a stationary and a moving object were detected and a pedestrian as well as a vehicle was detected.

Depth-based target segmentation for intelligent vehicles: fusion of radar and binocular stereo

Dynamic environment interpretation is of special interest for intelligent vehicle systems. It is expected to provide lane information, target depth, and the image positions of targets within given depth ranges. Typical segmentation algorithms cannot solve the problems satisfactorily, especially under the high-speed requirements of a real-time environment. Furthermore, the variation of image positions and sizes of targets creates difficulties for tracking. In this paper, we propose a sensor-fusion method that can make use of coarse target depth information to segment target locations in video images. Coarse depth ranges can be provided by radar systems or by a vision-based algorithm introduced in the paper. The new segmentation method offers more accuracy and robustness while decreasing the computational load.

Cortical hypercolumn size determines stereo fusion limits.

The size of a pair of cortical ocular dominance columns determines a basic anatomical module of V-1 which Hubel and Wiesel have termed the hypercolumn. Does this correspond to a basic functional, or psychophysically measurable, module as well? This is the basic question addressed in the present paper. Since the ocular dominance column architecture is presumed to be related to stereo vision, it is natural to assume that hypercolumn size should provide a modular basis for basic phenomena of stereopsis. In previous work, we have suggested that local nonlinear filtering via the cepstral transform, operating on a local window of cortical tissue scaled by hypercolumn size, provides such a modular model of stereopsis. In the present paper, we review this model and then discuss a number of issues related to the biological plausibility and implementation of this algorithm. Then, we present the main result of this paper: we have analyzed a number of experiments related to stereo fusion limits (Panum's area) and to disparity gradient and disparity scaling, and demonstrate that there is a simple unifying explanation for these phenomena in terms of a constant cortical module whose size is determined by a pair of ocular dominance columns. As a corollary, Panum's area must increase according to (inverse) cortical magnification factor. We show that this is supported by all existing experimental data.

RECIPROCAL-WEDGE TRANSFORM IN ACTIVE STEREO

The Reciprocal-Wedge Transform (RWT) facilitates space-variant image representation. In this paper a V-plane projection method is presented as a model for imaging using the RWT. It is then shown that space-variant sensing with this new RWT imaging model is suitable for fixation control in active stereo that exhibits vergence and versional eye movements and scanpath behaviors. A computational interpretation of stereo fusion in relation to disparity limit in space-variant imagery leads to the development of a computational model for binocular fixation. The vergence-version movement sequence is implemented as an effective fixation mechanism using the RWT imaging. A fixation system is presented to show the various modules of camera control, vergence and version.