High-speed 3D sensing via hybrid-mode imaging and guided upsampling

Jonathan Leach,Feng Zhu,Max Tyler,Sam W. Hutchings,Abderrahim Halimi,Stephen McLaughlin,Susan Chan,Robert K. Henderson,Istvan Gyongy

doi:10.1364/optica.390099

Abstract

Imaging systems with temporal resolution play a vital role in a diverse range of scientific, industrial, and consumer applications, e.g., fluorescent lifetime imaging in microscopy and time-of-flight (ToF) depth sensing in autonomous vehicles. In recent years, single-photon avalanche diode (SPAD) arrays with picosecond timing capabilities have emerged as a key technology driving these systems forward. Here we report a high-speed 3D imaging system enabled by a state-of-the-art SPAD sensor used in a hybrid imaging mode that can perform multi-event histogramming. The hybrid imaging modality alternates between photon counting and timing frames at rates exceeding 1000 frames per second, enabling guided upscaling of depth data from a native resolution of 64 × 32 to 256 × 128 . The combination of hardware and processing allows us to demonstrate high-speed ToF 3D imaging in outdoor conditions and with low latency. The results indicate potential in a range of applications where real-time, high throughput data are necessary. One such example is improving the accuracy and speed of situational awareness in autonomous systems and robotics.

Highlights

Three-dimensional depth sensing is used in a growing range of applications, including autonomous vehicles [1], industrial machine vision [2], gesture recognition in computer interfaces [3], and augmented and virtual reality [4]
Whilst frame rates of 10-60 frames per second (FPS) are typical for ToF, an order of magnitude faster acquisition rates, coupled with minimal latency would be beneficial in several applications
The advantage of the hybrid imaging mode is that we have a high resolution intensity image with which to guide the upsampling of the lower resolution depth information, resulting in a four-fold improvement in the spatial resolution of the depth data. The sensor operates such that alternating frames at ≈ 500 FPS in intensity and histogram mode are captured, providing an overall frame rate of ≈ 1 kFPS

Summary

INTRODUCTION

Three-dimensional depth sensing is used in a growing range of applications, including autonomous vehicles [1], industrial machine vision [2], gesture recognition in computer interfaces [3], and augmented and virtual reality [4]. The advantage of the hybrid imaging mode is that we have a high resolution intensity image with which to guide the upsampling of the lower resolution depth information, resulting in a four-fold improvement in the spatial resolution of the depth data The sensor operates such that alternating frames at ≈ 500 FPS in intensity and histogram mode are captured, providing an overall frame rate of ≈ 1 kFPS. The work presented here demonstrates high-speed 3D imaging in ambient light conditions This is enabled by the unique combination of the factors mentioned above: first, the stateof-the-art SPAD array that can operate in a high photon flux environment; second, firmware that enables alternating modes of imaging at high rates; and third, the guided upsampling algorithm that upscales the native resolution of the depth data. We note that as the SPADs are continually active, rather being turned on at the start of the timing period, detector pile-up due to the SPAD dead-time and macro-pixel combination tree [25] does not distort towards early time bins either

EXPERIMENT

Background

DATA ANALYSIS

RESULTS

DISCUSSION AND OUTLOOK