A video inpainting method for unmanned vehicle based on fusion of time series optical flow information and spatial information

Huayan Pu,Shaorong Xie,Rui Zhao,Jun Luo,Jingyi Liu,Hengyu Li

doi:10.1177/17298814211053103

Abstract

In this article, the problem of video inpainting combines multiview spatial information and interframe information between video sequences. A vision system is an important way for autonomous vehicles to obtain information about the external environment. Loss or distortion of visual images caused by camera damage or pollution seriously makes an impact on the vision system ability to correctly perceive and understand the external environment. In this article, we solve the problem of image restoration by combining the optical flow information between frames in the video with the spatial information from multiple perspectives. To solve the problems of noise in the single-frame images of video frames, we propose a complete two-stage video repair method. We combine the spatial information of images from different perspectives and the optical flow information of the video sequence to assist and constrain the repair of damaged images in the video. This method combines the interframe information of the front and rear image frames with the multiview image information in the video and performs video repair based on optical flow and a conditional generation adversarial network. This method regards video inpainting as a pixel propagation problem, uses the interframe information in the video for video inpainting, and introduces multiview information to assist the repair based on a conditional generative adversarial network. This method was trained and tested in Zurich using a data set recorded by a pair of cameras mounted on a mobile platform.

Highlights

Due to the abundant image information provided by the multiview system, the multiview image acquisition system is widely used in navigation,[3] panorama,[4] the occlusion process and vehicle classification,[5] target detection,[6,7,8] and tracking.[9,10,26]
Positioning is used for assisted driving systems, and multiview vision systems are used for obtaining surround views around the car body
We combine the network model based on the conditional generation adversarial network and the optical flow information in the video sequence

Summary

Introduction

With the development of deep learning and computer vision technology, multiview vision system has become significant in the field of automation.[1,2] Due to the abundant image information provided by the multiview system, the multiview image acquisition system is widely used in navigation,[3] panorama,[4] the occlusion process and vehicle classification,[5] target detection,[6,7,8] and tracking.[9,10,26] As shown in Figure 1, multiview vision target recognition and. Most of the pixels in the missing area can be made up by spreading from the visible area This method can make full use of the interframe information of the images in the video sequence, and it is far easier to repair the single-frame image in the video sequence through the refinement of the optical flow field. The second stage utilizes the initial inpainting results from the first part to extract the optical flow of the video sequence and completes the refinement and inpainting process of the optical flow field from coarse to fine through the deep convolutional network. Damaged image frames in a single-view video sequence can get accurate and clear results by this method that fuses spatial transformation, group convolution and features channel information exchange. This condition means that after the pixel has propagated forward and backward, the pixel should return to its original position

Experiments

Method

Conclusion