Learning Optical Flow Using Deep Dilated Residual Networks

Abstract

Nowadays, convolutional neural networks achieve remarkable performance on optical flow estimation because of its strong non-linear fitting ability. Most of them adopt the U-Net architecture, which contains an encoder part and a decoder part. In the encoder part, the resolution of the feature map is reduced with the deepening of the network layer. In the decoder part, the feature map is enlarged by the deconvolution layer to recover the estimated flow as full resolution. However, the motion details are usually lost with the contracting and expanding operations. Moreover, learning methods, especially supervised networks, always ignore the advantages of many well-proven constraints used in the variational model. In this paper, we introduce a novel architecture named dilated residual networks for learning optical flow, which can avoid the loss of details of the U-Net architecture and can directly learn the residual functions rather than the unreferenced functions to enhance the learning ability of the network. Furthermore, inspired by variational methods, the traditional prior assumptions, such as brightness constancy, gradient constancy, and smoothness assumption, are used in the supervised network as extra auxiliary terms to guide the training of network. Our method is tested on several benchmarks, such as MPI-Sintel, KITTI2012, and KITTI2015. The experimental results show that the dilated residual network is suitable for dense optical flow estimation due to the capability of preserving motion details and can boost the accuracy of optical flow estimation.