Improved Discrete Optical Flow Estimation With Triple Image Matching Cost

Fei Yang,Yongmei Cheng,Joost Van De Weijer,Mikhail G Mozerov

doi:10.1109/access.2020.2968180

Abstract

Approaches that use more than two consecutive video frames in the optical flow estimation have a long research history. However, almost all such methods utilize extra information for a pre-processing flow prediction or for a post-processing flow correction and filtering. In contrast, this paper differs from previously developed techniques. We propose a new algorithm for the likelihood function calculation (alternatively the matching cost volume) that is used in the maximum a posteriori estimation. We exploit the fact that in general, optical flow is locally constant in the sense of time and the likelihood function depends on both the previous and the future frame. Implementation of our idea increases the robustness of optical flow estimation. As a result, our method outperforms 9% over the DCFlow technique, which we use as prototype for our CNN based computation architecture, on the most challenging MPI-Sintel dataset for the non-occluded mask metric. Furthermore, our approach considerably increases the accuracy of the flow estimation for the matching cost processing, consequently outperforming the original DCFlow algorithm results up to 50% in occluded regions and up to 9% in non-occluded regions on the MPI-Sintel dataset. The experimental section shows that the proposed method achieves state-of-the-arts results especially on the MPI-Sintel dataset.

Highlights

Optical flow estimation is important for a large variety of computer vision applications, such as 3D scene reconstruction, autonomous driving systems and robotics
We propose a new matching cost formation based on two assumptions: most occlusion regions that are invisible in the forward frame image are visible in the backward frame; the forward flow is approximately equal to the negative value of the backward flow
PROBLEM DEFINITION Discrete optical flow estimation belongs to the general matching problem, and in the framework of the global approach the matching problem is formulated in terms of energy minimization with the energy function in the following form: E (v) = Cp vp +

Summary

INTRODUCTION

Optical flow estimation is important for a large variety of computer vision applications, such as 3D scene reconstruction, autonomous driving systems and robotics. All methods that use more than two images can be considered as related work, the proposed triple patch match model is fundamentally different from these approaches. Janai et al [41] proposed an unsupervised learning method for multi-frame optical flow They construct past cost volume and future cost volume with three frames and leverage convolutional neural network to reason occlusion. We propose a new matching cost formation based on two assumptions: most occlusion regions that are invisible in the forward frame image (relative to the current frame) are visible in the backward frame; the forward flow is approximately equal to the negative value of the backward flow. Our approach considerably increases the optical flow estimation on the MPI-Sintel dataset [44] after the matching cost processing that is the most important part of the proposed pipeline.

PROBLEM DEFINITION

IMPROVED OPTICAL FLOW ESTIMATION PIPELINE

EXPERIMENTAL RESULTS

CONCLUSION