Spacetime Stereo Research Articles

Booster: A Benchmark for Depth From Images of Specular and Transparent Surfaces.

Estimating depth from images nowadays yields outstanding results, both in terms of in-domain accuracy and generalization. However, we identify two main challenges that remain open in this field: dealing with non-Lambertian materials and effectively processing high-resolution images. Purposely, we propose a novel dataset that includes accurate and dense ground-truth labels at high resolution, featuring scenes containing several specular and transparent surfaces. Our acquisition pipeline leverages a novel deep space-time stereo framework, enabling easy and accurate labeling with sub-pixel precision. The dataset is composed of 606 samples collected in 85 different scenes, each sample includes both a high-resolution pair (12 Mpx) as well as an unbalanced stereo pair (Left: 12 Mpx, Right: 1.1 Mpx), typical of modern mobile devices that mount sensors with different resolutions. Additionally, we provide manually annotated material segmentation masks and 15 K unlabeled samples. The dataset is composed of a train set and two test sets, the latter devoted to the evaluation of stereo and monocular depth estimation networks. Our experiments highlight the open challenges and future research directions in this field.

Single-stripe-enhanced spacetime stereo reconstruction for concrete defect identification

Few studies explored the 3D reconstruction of concrete defects, and existing methods often suffer from noise interference or impracticality in real-world applications. This study proposes a novel single-stripe-enhanced spacetime stereo reconstruction method that combines a laser liner for structured light generation and a stereo vision camera. This method analyses the morphology and intensity variation of laser stripes from images captured from single scans, achieving accurate defect reconstructions. It is designed for handheld scenarios with image re-alignment eliminating the vibration caused by handheld operation. The morphology of laser stripes at crack sections is analysed for robust fine crack reconstruction. Active frame searching is introduced to improve the calculation efficiency, enabling spacetime stereo to operate over hundreds of frames. A comparative study of the proposed method and two commonly-used stereo matching methods demonstrates its ability to mitigate the impact of handheld operation and generate 3D reconstructions that highlight potential crack regions.

A Comprehensive Performance Evaluation of 3-D Transformation Estimation Techniques in Point Cloud Registration

3-D local feature extraction and matching is a key step in point cloud registration. However, this process commonly draws into false correspondences caused by noise, occlusion, incomplete surface, and so on. To estimate correct transformation based on these corrupted correspondences, numerous transformation estimation techniques have been proposed. However, no comprehensive study comparing their accuracy, robustness, and efficiency performance under different nuisances has been conducted. This article evaluates thirteen popular transformation estimation proposals on both descriptor-based and synthetic correspondences. On descriptor-based correspondences, comprehensive evaluation items (e.g., combining with iterative closest point (ICP) and different local features) of these methods are tested on five popular datasets acquired with different devices (e.g., Minolta vivid scanner, Microsoft Kinect, and Space Time Stereo). On synthetic correspondences, the robustness of these methods to varying percentages of correct correspondences (PCCs) is evaluated. In addition, their efficiency is also evaluated. The results present some valuable findings that may provide a supplement to existing evaluations of transformation estimation techniques. A summary of merits, demerits, and application guidance of these tested methods is finally presented to guide real-world applications and new transformation estimation techniques crafting.

Spacetime Stereo and 3D Flow via Binocular Spatiotemporal Orientation Analysis.

This paper presents a novel approach to recovering estimates of 3D structure and motion of a dynamic scene from a sequence of binocular stereo images. The approach is based on matching spatiotemporal orientation distributions between left and right temporal image streams, which encapsulates both local spatial and temporal structure for disparity estimation. By capturing spatial and temporal structure in this unified fashion, both sources of information combine to yield disparity estimates that are naturally temporal coherent, while helping to resolve matches that might be ambiguous when either source is considered alone. Further, by allowing subsets of the orientation measurements to support different disparity estimates, an approach to recovering multilayer disparity from spacetime stereo is realized. Similarly, the matched distributions allow for direct recovery of dense, robust estimates of 3D scene flow. The approach has been implemented with real-time performance on commodity GPUs using OpenCL. Empirical evaluation shows that the proposed approach yields qualitatively and quantitatively superior estimates in comparison to various alternative approaches, including the ability to provide accurate multilayer estimates in the presence of (semi)transparent and specular surfaces.

Space-time stereo analysis combining local structure and modulation features in the monogenic wavelet domain

A multimodal cost function, based on the local multimodal image descriptors that combine local structure features (orientation, coherency) and modulation/localization information (amplitude, phase, and spatial frequency) of the monogenic wavelet transform, is proposed to estimate the time-varying disparity maps in the space-time stereo framework. The proposed cost function makes use of a constraint of local orientation, phase, and amplitude congruencies with the weighted coefficients, which are adapted to local image features and are insusceptible to level shift, scaling, and rotation and lighting invariance. Experiments on the synthetic and natural stereo sequences show the estimated results are more robust than the intensity-based sum of standard sum of squared difference cost function.

Scalable 3D video of dynamic scenes

In this paper we present a scalable 3D video framework for capturing and rendering dynamic scenes. The acquisition system is based on multiple sparsely placed 3D video bricks, each comprising a projector, two grayscale cameras, and a color camera. Relying on structured light with complementary patterns, texture images and pattern-augmented views of the scene are acquired simultaneously by time-multiplexed projections and synchronized camera exposures. Using space–time stereo on the acquired pattern images, high-quality depth maps are extracted, whose corresponding surface samples are merged into a view-independent, point-based 3D data structure. This representation allows for effective photo-consistency enforcement and outlier removal, leading to a significant decrease of visual artifacts and a high resulting rendering quality using EWA volume splatting. Our framework and its view-independent representation allow for simple and straightforward editing of 3D video. In order to demonstrate its flexibility, we show compositing techniques and spatiotemporal effects.

Spacetime stereo: a unifying framework for depth from triangulation

Depth from triangulation has traditionally been investigated in a number of independent threads of research, with methods such as stereo, laser scanning, and coded structured light considered separately. In this paper, we propose a common framework called spacetime stereo that unifies and generalizes many of these previous methods. To show the practical utility of the framework, we develop two new algorithms for depth estimation: depth from unstructured illumination change and depth estimation in dynamic scenes. Based on our analysis, we show that methods derived from the spacetime stereo framework can be used to recover depth in situations in which existing methods perform poorly.

Spacetime faces

We present an end-to-end system that goes from video sequences to high resolution, editable, dynamically controllable face models. The capture system employs synchronized video cameras and structured light projectors to record videos of a moving face from multiple viewpoints. A novel spacetime stereo algorithm is introduced to compute depth maps accurately and overcome over-fitting deficiencies in prior work. A new template fitting and tracking procedure fills in missing data and yields point correspondence across the entire sequence without using markers. We demonstrate a data-driven, interactive method for inverse kinematics that draws on the large set of fitted templates and allows for posing new expressions by dragging surface points directly. Finally, we describe new tools that model the dynamics in the input sequence to enable new animations, created via key-framing or texture-synthesis techniques.