Learning Super-Resolution Reconstruction for High Temporal Resolution Spike Stream

Abstract

Spike camera is a new type of bio-inspired vision sensor, each pixel of which perceives the brightness of the scene independently, and finally outputs 3-dimensional spatiotemporal spike streams. To bridge the spike camera and traditional frame-based vision, there is some works to reconstruct spike streams into regular images. However, the low spatial resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$400\times 250$ </tex-math></inline-formula> ) of the spike camera limits the quality of the reconstructed images. Thus, it is meaningful to explore a super-resolution reconstruction for spike streams. In this paper, we propose an end-to-end network to reconstruct high-resolution images from low-resolution spike streams. To utilize more spatiotemporal features of spike streams, our network adopts a multi-level features learning mechanism, including intra-stream feature extraction by spike encoder, inter-stream dependencies extraction based on optical flow module, and joint features learning via spike-based iterative projection. Experimental results demonstrate that our network is superior to the combination of state-of-the-art intensity image reconstruction methods and super-resolution networks on simulated and real datasets.