Real-time object detection in 360-degree videos

Abstract

Streaming of 360-degree videos over the internet is challenging task, but it provides rich multimedia experiences by allowing viewers to navigate 360-degree contents. The 360-degree videos need larger bandwidth and less latency to be streamed over the internet than the conventional videos. Therefore, non-visible area must be discarded from the video to save bandwidth. View prediction techniques have been used to predict visible area of the 360-degree video frames to be streamed. Linear regression using viewer’s past viewing behavior data is useful to predict short-term future behavior of the viewer, which is not useful when the network delay is longer than the prediction horizon. Object detection techniques help predicting viewers’ future motion for longer prediction horizon since the viewers tend to follow the objects that draw their attention. However, conventional object detection techniques using a convolutional neural network, such as YOLO, are difficult to be applied to 360-degree videos. There are distortions in the 360-degree videos when the spherical 360-degree video is projected into equi-rectangular videos for processing and storing purposes. A same object could have different shapes in the equi-rectangular video depends on their angular position in the sphere. Therefore, in this paper, we propose a multi-directional projection (MDP) technique to detect objects in the 360-degree videos. The proposed multi- directional projection technique mitigates the distortions in the equi-rectangular videos and feeds the redirected videos to the object detection system. Therefore, the neural network trained with conventional video dataset can be used without any change. Experimental result shows that the proposed method helps detecting objects in the edges of the 360-degree videos.