Video object segmentation using spatio-temporal deep network

Abstract

Video analysis is increasingly becoming possible with improvement in hardware and deep learning algorithms. Videos contain the spatial as well as the temporal information that come closest to the real-world visual information representation. Albeit the human brain can make better decisions using spatio-temporal data, the images and video frames captured from the same standard RGB camera will vary in quality. Deep learning has resulted in extraordinary performances for image analysis. Image-based deep networks have been modified and extended to work on video, and optical flow between the frames has been utilized to capture temporal variations. There is a gap in understanding whether such networks capture the spatio-temporal information collectively. The network that can capture the information effectively should be capable of good performances despite relatively bad quality video frames. In this work, different deep network architectures are explored and their ability to capture spatio-temporal features is explored. With the understanding of the advantages and disadvantages of the network components, a new network is designed for the task of video object segmentation (VOS). The performance of the proposed network is evaluated using the DAVIS dataset for three tasks: VOS using weak supervision, zero-shot VOS and one-shot VOS. The best performance is reported in comparison to the state-of-the-art on DAVIS dataset and the robustness of the model to noisy labels is demonstrated.