Benchmarking equivariance for Deep Learning based optical flow estimators

Abstract

The Optical Flow (OF) estimation is an important but challenging task used in various computer vision applications. Recently, the introduction of Deep Learning models has resulted in a veritable paradigm shift. However, such an approach can be biased towards the specific training data, which can lead to a considerable inaccuracy of the estimated OF in real world applications. This paper proposes a novel benchmark that can be used to identify and measure different biases of Deep Learning models for OF estimation. The benchmark focuses on testing the network equivariance, i.e. the model capability to handle data transformations. We have performed experiments based on public datasets to (1) investigate to what extent the state-of-the-art networks lack spatial equivariance when reflections are applied to the data; (2) propose new metrics and a methodology to assess the phenomenon; and (3) benchmark the state-of-the-art optical estimators and their core components for equivariance. The results show that some state-of-the-art Deep Learning techniques present a substantial degree of bias towards certain directions of motion. The proposed framework can help researchers and practitioners to develop more effective models for OF estimation. The testing and training scripts provided at https://github.com/stsavian/benchmarking_equivariance_for_of_estimators can be used to evaluate and compare different models for OF estimation.