3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing

Ling Bai,Ming Cen,Fangchao Hu,Yinguo Li

doi:10.3390/rs13163288

Abstract

Since single sensor and high-density point cloud data processing have certain direct processing limitations in urban traffic scenarios, this paper proposes a 3D instance segmentation and object detection framework for urban transportation scenes based on the fusion of Lidar remote sensing technology and optical image sensing technology. Firstly, multi-source and multi-mode data pre-fusion and alignment of Lidar and camera sensor data are effectively carried out, and then a unique and innovative network of stereo regional proposal selective search-driven DAGNN is constructed. Finally, using the multi-dimensional information interaction, three-dimensional point clouds with multi-features and unique concave-convex geometric characteristics are instance over-segmented and clustered by the hypervoxel storage in the remarkable octree and growing voxels. Finally, the positioning and semantic information of significant 3D object detection in this paper are visualized by multi-dimensional mapping of the boundary box. The experimental results validate the effectiveness of the proposed framework with excellent feedback for small objects, object stacking, and object occlusion. It can be a remediable or alternative plan to a single sensor and provide an essential theoretical and application basis for remote sensing, autonomous driving, environment modeling, autonomous navigation, and path planning under the V2X intelligent network space– ground integration in the future.

Highlights

With the rapid development of artificial intelligence and automation, an intelligent fusion system with top-down and multi-tier architecture is constructed by multiple vehicular sensors, combining environmental perception, path planning, intelligent behavior decisionmaking, automatic control, and vehicle architecture
It can be seen that the network structure of this paper takes into account over-segmentation
It can be seen that the network structure of this paper takes into acthe regional proposal of the of stereo imageimage pair aspair wellasaswell the dilated convolution and double count the regional proposal the stereo as the dilated convolution and integrated loss function to expand the view field of calculation, so it has a certain double integrated loss function to expand the view field of calculation, so it has adetection certain effect gain and gain compensation amount for small for objects, and occlusion

Summary

Introduction

With the rapid development of artificial intelligence and automation, an intelligent fusion system with top-down and multi-tier architecture is constructed by multiple vehicular sensors, combining environmental perception, path planning, intelligent behavior decisionmaking, automatic control, and vehicle architecture. As an essential basis for realizing self-driving and safe driving, the perception of driving environment information around intelligent vehicles has always been of tremendous application challenge and theoretical research value. The application of multi-sensor data fusion has gradually revealed its advantages in intelligent terminal equipment and augmented reality technology [1]. It is challenging to obtain complex three-dimensional (3D) spatial information using the two-dimensional image information acquired by the visual sensor, and the computational cost is expensive. Even if the same object is regarded as an invariant, the calculation result of its segmentation will be affected by many variables such as illuminant source and noise. Even if the same object is regarded as an invariant, the calculation result of its segmentation will be affected by many variables such as illuminant source and noise. 4.0/).

Objectives

Results

Discussion

Conclusion