Real-Time Stereo Disparity Prediction Based on Patch-Embedded Extraction and Depthwise Hierarchical Refinement for 3-D Sensing of Autonomous Vehicles on Energy-Efficient Edge Computing Devices

Abstract

Sensing and communications are dispensable for autonomous vehicles and IoT. One key task in autonomous driving is the sensing of 3D information surrounding a vehicle. Most existing stereo disparity prediction networks pursue accurate disparity maps on high-performance GPUs with high energy consumption. While very few stereo networks achieving real-time on resource-constrained edge devices are hardly satisfactory in terms of accuracy. To tackle this, we propose a lightweight and efficient three-stage stereo disparity prediction network named HRSNet for real-time stereo matching on energy-efficient edge devices with limited resource, reducing energy consumption and requirement on high-performance computing resources. The network consists of a serial patch-embedded feature extractor with short shortcut connections and a depthwise hierarchical refinement. In the stage of refining disparity maps, we employ a novel 2D aggregation network containing expanded depthwise separable convolutions with residual connections to regularize 3D group-wise cost volumes, resulting in accuracy improvement and inference acceleration. By evaluating on KITTI 2015, the proposed network achieves approaching results to state-of-the-art metrics on high-end GPU, and frame-rate@D1-all results of 64.6 FPS@6.16%, 41.7 FPS@3.49%, 32.2 FPS@2.29% in Stages 1-3 on Jetson Nano with ultra-low average power consumption of 6.3 W after tensorRT optimization.