Real-time depth estimation using camera and IMU on the unstabilized platform of a spherical robot

Abstract

Vision based robot navigation relies on the image sequences that are captured by the camera attached to its platform. In many robotic applications such as in case of spherical robots used for surveillance, the platform on which the camera resides is often unsteady and unwanted relative motion exists between camera and scene. This unwanted relative motion in case of a spherical robot is due to the pitching motion associated with its platform called yoke. A depth estimation algorithm that handles the effect of pitching using non-linear observer approach is proposed. The object in the scene whose depth is to be estimated is detected as features in the acquired images and are tracked by using concepts of optical flow. A discrete time state space model that fuses the information from camera and IMU data attached to the unsteady platform of spherical robot is derived. Extended Kalman filter (EKF) is used as the non-linear estimation technique for extraction of depth information from the proposed state space model. The convergence aspects of the extended Kalman filter when used as a deterministic observer for the proposed non-linear discrete-time model is analyzed with local observability and it is shown that there is boundedness of error covariance between the observed and actual depth. It is shown that the estimation error converges to zero irrespective of the initialization provided to the observer.