Monocular, Boundary-Preserving Joint Recovery of Scene Flow and Depth

Yosra Mathlouthi,Amar Mitiche,Ismail Ben Ayed

doi:10.3389/fict.2016.00021

Yosra Mathlouthi, Amar Mitiche + Show 1 more

Open Access

https://doi.org/10.3389/fict.2016.00021

Copy DOI

Abstract

Variational joint recovery of scene flow and depth from a single image sequence, rather than from a stereo sequence as others required, was investigated in Mitiche et al. (2015) using an integral functional with a term of conformity of scene flow and depth to the image sequence spatiotemporal variations, and L2 regularization terms for smooth depth field and scene flow. The resulting scheme was analogous to the Horn and Schunck optical flow estimation method except that the unknowns were depth and scene flow rather than optical flow. Several examples were given to show the basic potency of the method: It was able to recover good depth and motion, except at their boundaries because L2 regularization is blind to discontinuities which it smooths indiscriminately. The method we study in this paper generalizes to L1 regularization the formulation of Mitiche et al. (2015) so that it computes boundary preserving estimates of both depth and scene flow. The image derivatives, which appear as data in the functional, are computed from the recorded image sequence also by a variational method which uses L1 regularization to preserve their discontinuities. Although L1 regularization yields nonlinear Euler-Lagrange equations for the minimization of the objective functional, these can be solved efficiently. The advantages of the generalization, namely sharper computed depth and three-dimensional motion, are put in evidence in experimentation with real and synthetic images which shows the results of L1 versus L2 regularization of depth and motion, as well as the results using L1 rather than L2 regularization of image derivatives.

Highlights

Scene flow is the three-dimensional (3D) motion field of the visible environmental surfaces projected on the image domain: at each image point, scene flow is the 3D velocity of the corresponding environmental surface point
The scheme improved significantly on others because it needed a single image sequence rather than a stereo stream like other methods. It formulated the problem using an integral functional of two terms: a data fidelity term to constrain the computed scene flow to conform to the image sequence spatiotemporal derivatives and L2 regularization terms to constrain the computed scene flow and depth to be smooth, which led to linear Euler–Lagrange equations for the minimization of the objective functional, much like in the Horn and Schunck optical flow formulation (Horn and Schunk, 1981), except that it involved depth and scene flow rather than optical flow
We show comparative results that put in evidence the improvements one can obtain by using L1 rather than L2 regularization of depth and motion, as well as by evaluating the image derivatives with L1 rather than L2 regularization

Summary

Introduction

Scene flow is the three-dimensional (3D) motion field of the visible environmental surfaces projected on the image domain: at each image point, scene flow is the 3D velocity of the corresponding environmental surface point. Basic variational statements use L2 (Tikhonov) regularization This regularization, which imposes smoothness on the solution, yields linear terms in the Euler–Lagrange equations. The scheme improved significantly on others because it needed a single image sequence rather than a stereo stream like other methods It formulated the problem using an integral functional of two terms: a data fidelity term to constrain the computed scene flow to conform to the image sequence spatiotemporal derivatives and L2 regularization terms to constrain the computed scene flow and depth to be smooth, which led to linear Euler–Lagrange equations for the minimization of the objective functional, much like in the Horn and Schunck optical flow formulation (Horn and Schunk, 1981), except that it involved depth and scene flow rather than optical flow. The variation of motion and structure in such a case can be sharp and significant at these objects occluding boundaries, and accuracy would require that they be preserved by the regularization operator

Methods

Results

Conclusion