Differential SfM and image correction for a rolling shutter stereo rig

Abstract

• A simple and flexible generalized RS stereo differential SfM algorithm over two consecutive frames. • Two effective and effcient RS-stereo non-linear optimization techniques based on the maximum likelihood criterion. • An RS-stereo image correction method to remove the inaccuracies induced by the RS effect. • The proposed model and method are universal and tractable. Most modern consumer-grade cameras are equipped with an electronic rolling shutter (RS), leading to image distortions when the camera moves during image acquisition. We explore the first structure and motion estimation problem of a dynamic generalized RS stereo camera. Such a general configuration is commonplace in robots and autonomous driving applications. We propose a tractable RS stereo differential structure from motion (SfM) algorithm, taking into account the RS effect during consecutive imaging, which effectively compensates for the RS-stereo image distortion by a linear scaling operation on each optical flow. We further propose embedding the cheirality into RANSAC and develop a robust RS-stereo-aware full-motion estimation framework. We demonstrate that the RS stereo motion and depth map refined by our non-linear optimization schemes within the maximum likelihood criterion can be used for image correction to recover high-quality global shutter (GS) stereo images. Moreover, using the proposed generalized RS stereo differential SfM pipeline, the corrected images produce an accurate 3D scene structure as the ground-truth structure. Extensive experiments on both synthetic and real RS stereo data demonstrate the effectiveness of our model and method in various configurations.