Frame interpolation in video stream using optical flow methods

Abstract

The article deals with the issues of interpolation and extrapolation of video stream frames in real time. Results of research of the method of generation of intermediate frames of a video stream are given. The method is based on the optical flow calculate. The results of comparison of implementations of the method using the Farneback, Brox and Duality based TV-L1 approaches for the calculation of optical flow and CUDA technology are presented. It is shown that the Farneback method best result is gives. The Duality based TV-L1 method shows a smooth and high-quality optical flow, but gives a very slow calculation speed. The Brox method is not suitable at all, since it shows the worst result both in terms of calculation speed and optical flow quality. The results of comparison of implementations based on CUDA and OpenCL technologies are also presented. OpenCL implementations are almost two times slower than CUDA implementation. This leads to the inability to generate frames in 60 FPS in real time even for low resolution images. However, CUDA can work only on NVidia GPU accelerators. In conclusion, the ways of further research on this topic based on the use of deep neural networks are presented.