The Data Fusion and Semantic Segmentation Method Based on Multi-Sensor System

Abstract

The design and implementation of a prototype environmental perception system are introduced, which consists of the multi-camera, LiDAR, and IMU sensor. The processing steps of the environmental data collected by the multi-sensor platform include joint calibration, data fusion, and semantic segmentation. First, the optimization method based on the epipolar constraint is proposed for the joint calibration of the multi-camera and the LiDAR system. In the data fusion section, an improved data association method for the point cloud is proposed where the foreground segmentation method is used to reduce scale estimation error due to the scale-sudden-change. Finally, the semantic segmentation method based on the deep-learning is proposed. The convolutional neural network based on the Squeeze-and-Excitation Net is designed to identify and classify the point cloud data accurately. The physical simulation result shows that the system collects and identifies the environmental information accurately.