3D Scene Flow Estimation with a Piecewise Rigid Scene Model

Christoph Vogel,Konrad Schindler,Stefan Roth

doi:10.1007/s11263-015-0806-0

Christoph Vogel, Konrad Schindler + Show 1 more

Open Access

https://doi.org/10.1007/s11263-015-0806-0

Copy DOI

Abstract

3D scene flow estimation aims to jointly recover dense geometry and 3D motion from stereoscopic image sequences, thus generalizes classical disparity and 2D optical flow estimation. To realize its conceptual benefits and overcome limitations of many existing methods, we propose to represent the dynamic scene as a collection of rigidly moving planes, into which the input images are segmented. Geometry and 3D motion are then jointly recovered alongside an over-segmentation of the scene. This piecewise rigid scene model is significantly more parsimonious than conventional pixel-based representations, yet retains the ability to represent real-world scenes with independent object motion. It, furthermore, enables us to define suitable scene priors, perform occlusion reasoning, and leverage discrete optimization schemes toward stable and accurate results. Assuming the rigid motion to persist approximately over time additionally enables us to incorporate multiple frames into the inference. To that end, each view holds its own representation, which is encouraged to be consistent across all other viewpoints and frames in a temporal window. We show that such a view-consistent multi-frame scheme significantly improves accuracy, especially in the presence of occlusions, and increases robustness against adverse imaging conditions. Our method currently achieves leading performance on the KITTI benchmark, for both flow and stereo.

Highlights

The scene flow of a dynamic scene is defined as a dense representation of the 3D shape and its 3D motion field
To estimate 3D scene flow, we describe the dynamic scene as a collection of piecewise planar regions moving rigidly over time (Fig. 3)
We model shape and motion priors independently, and define our regularizer E R(P, S ) as the sum of a geometric term and a term to measure the regularity of the motion field

Summary

Introduction

The scene flow of a dynamic scene is defined as a dense representation of the 3D shape and its 3D motion field. Applications that benefit from knowing the scene flow include 3D video generation for 3D-TV (Hung et al 2013), motion capture (Courchay et al 2009; Park et al 2012; Vedula et al 1999), and driver assistance (e.g., Müller et al 2011; Rabe et al 2010; Wedel et al 2008). The 3D scene flow can be seen as a combination of two classical computer vision problems—it generalizes optical flow to 3D, or alternatively, dense stereo to dynamic scenes. Methods emerged (Vogel et al 2013b, 2014; Yamaguchi et al 2014) that could leverage the additional information present in stereo video streams and outperform their dedicated twodimensional counterparts at their respective tasks

Objectives

Results

Conclusion