An ultra-high-speed hardware accelerator for image reconstruction and stereo rectification on event-based camera

Abstract

Event-based cameras are novel bio-inspired vision sensors, which sense brightness changes rather than the actual intensity level. In contrast to conventional cameras, such cameras capture new information about the scene in the form of sparse events at a very short latency. Recently, some researchers have studied on efficient event-based camera reconstruction approaches to obtain high-quality images. These efforts make performing stereo vision based on event-cameras being possible. However, traditional stereo architecture cannot process such high-speed image streams generated by the event-based camera in real-time (in the order of μs), which calls for new approaches.In this paper, we provide a set of practical solutions on visual image reconstruction and stereo rectification for spike camera, which serves two important procedures of stereo vision. We first provide FPGA-accelerated hardware architecture to achieve ultra-high-speed visual image reconstruction with full texture of natural scenes from spike data. Then, an FPGA-accelerated ultra-high-speed stereo rectification architecture is proposed to rectify the reconstructed images generated for stereo vision. In this architecture, a fully pipelined calculation module is designed to process the complex coordinate transformation operations, which puts in pipelined registers to maximize the clock frequency. To further improve the throughput, we propose a parallel processing architecture that uses multiple processing elements (PEs) to process multiple pixels per cycle. In addition, we design a memory management unit (MMU) to optimize the memory usage of the hardware resource. The whole architecture is effectively implemented on a Xilinx Zynq7100 FPGA chip. We evaluate the proposed architecture with different settings. The experiments show that our architecture can process the spike streams in real-time manner.