Estimating Optical Flow with Streaming Perception and Changing Trend Aiming to Complex Scenarios

Abstract

Most of the recent works on optical flow estimation focus on introducing more effective CNNs or even the Transformer architecture that is quite popular in today’s CV field, but they all still follow the traditional two-frame input structure. In this paper, we argue that optical flow estimation should be more focused on the dynamic information in the image sequence. To that end, we present a straightforward but effective streaming framework equipped with a novel prediction module for capturing optical flow trends, which is a point that has not yet been discussed in previous works. By employing the backbone of the strong baseline to extract the stream of features from three consecutive frames as inputs, this module can predict the current state and update the previous changing trend of optical flow for feature fusion at the next stage. Additionally, we redesign the training loss, providing two new cost functions, in particular, to measure the variations in the optical flow trend. At the forward stage, we deploy a historical buffer to maintain the inference speed so that it is unaffected by additional input frames. Compared with the strong baseline, our method improves the average end-point error by 21.12% on Sintel Clean and 12.94% on Sintel Final. In addition, it also outperforms existing methods in real-world scenarios with low brightness or fast-moving targets.