3D Object Detection Algorithm for Panoramic Images With Multi-Scale Convolutional Neural Network

Dianwei Wang,Zhijie Xu,Shiqian Wu,Yanhui He,Yongrui Qin,Ying Liu,Daxiang Li

doi:10.1109/access.2019.2955995

Abstract

This paper addresses the challenge of 3D object detection from a single panoramic image under severe deformation. The advent of the two-stage approach has impelled significant progress in 3D object detection. However, most available methods only can localize region proposals by a single-scale architecture network, which are sensitive to deformation and distortion. To address this issue, we propose a multi-scale convolutional neural network (MSCNN) to estimate the 3D pose of an object. To be specific, the proposed MSCNN consists of three steps for effectively detecting the distorted object on the panoramic images. The MSCNN contains the CycleGAN network that converts rectilinear images into panoramas, a fused framework that improves both accuracy and speed for object detection, and an adversarial spatial transformer network (ASTN) that extracts the deformation features of the object on panoramic images. Additionally, we recover the 3D pose of the object using a coordinate projection and a 3D bounding box. Extensive experiments demonstrate that the proposed method can achieve a 3D detection accuracy of 38.7% in high-resolution panoramic images, which is higher than the current state-of-the-art algorithm of 5.2%. Moreover, the speed of detection is only about 0.6 seconds per image, which is six times faster than Faster R-CNN (COCO). The code will be available at https://github.com/Yanhui-He.

Highlights

The panoramic image visualization platform has enjoyed popularity in many applications, such as virtual reality, visual surveillance, autonomous vehicles and virtual interaction [1]
We propose a new method to detect the objects of 360◦ panoramic imagery using a multi-scale convolutional neural network (MSCNN)
We propose a novel fusion network for 3D object detection with a Multi-scale Convolutional Neural Network (MSCNN), which learns a distortional representation for robust 3D detection localization in panoramic images

Summary

Introduction

The panoramic image visualization platform has enjoyed popularity in many applications, such as virtual reality, visual surveillance, autonomous vehicles and virtual interaction [1]. Panoramic images are typically represented using an equirectangular projection, The associate editor coordinating the review of this manuscript and approving it for publication was Li He. which creates severe geometric distortions for objects that are further from the central horizontal line [3]. Which creates severe geometric distortions for objects that are further from the central horizontal line [3] ERA images create new challenges for computer vision and image processing as i) we lack high-quality annotated 360◦ datasets, ii) imagery is difficult to treat due to its high-resolution and iii) equirectangular projection creates severe geometric distortions for objects away from the central horizontal line [3]. Panoramic images create new challenges for object detection, which is a crucial procedure

Methods

Results

Conclusion