Image Deblurring Aided by Low-Resolution Events

Zhouxia Wang,Jimmy Ren,Ping Luo,Jiawei Zhang

doi:10.3390/electronics11040631

Zhouxia Wang, Jimmy Ren + Show 2 more

Open Access

https://doi.org/10.3390/electronics11040631

Copy DOI

Abstract

Due to the limitation of event sensors, the spatial resolution of event data is relatively low compared to the spatial resolution of the conventional frame-based camera. However, low-spatial-resolution events recorded by event cameras are rich in temporal information which is helpful for image deblurring, while intensity images captured by frame cameras are in high resolution and have potential to promote the quality of events. Considering the complementarity between events and intensity images, an alternately performed model is proposed in this paper to deblur high-resolution images with the help of low-resolution events. This model is composed of two components: a DeblurNet and an EventSRNet. It first uses the DeblurNet to attain a preliminary sharp image aided by low-resolution events. Then, it enhances the quality of events with EventSRNet by extracting the structure information in the generated sharp image. Finally, the enhanced events are sent back into DeblurNet to attain a higher quality intensity image. Extensive evaluations on the synthetic GoPro dataset and real RGB-DAVIS dataset have shown the effectiveness of the proposed method.

Highlights

In contrast to the conventional frame-based camera which represents a dynamic scenario with a sequence of still images, an event-based vision sensor [1,2,3,4] tends to detect per-pixel brightness changes in microsecond resolution
This paper aimed to attain a blur-less and high-resolution intensity image with a sequence of low-spatial-resolution events recorded by an event camera and a high-spatialresolution blurry intensity image captured by a frame-based camera
DeblurNet is used for deblurring with the help of events data that own temporal information and an EventSR network (EventSRNet) used for enhancing event data assisted by high spatial resolution intensity images with enriched structure information

Summary

Introduction

In contrast to the conventional frame-based camera which represents a dynamic scenario with a sequence of still images, an event-based vision sensor [1,2,3,4] tends to detect per-pixel brightness changes in microsecond resolution. Once the logarithm of the intensity changes exceeds a preset threshold c in a given pixel (x, y), an event will be triggered which can be formulated as log(Ixy (t) + b) − log(Ixy (t − ∆t) + b) = p · c (1). Where Ixy (t) and Ixy (t − ∆t) denote the intensities at time t and t − ∆t, respectively. P ∈ {+1, −1} is the polarity representing the direction (increase or decrease) of the intensity change. Since event data are discrete and own high temporal resolution, a high dynamic range, and a low motion blur, the event camera has shown its potential in several robotic and computer vision tasks, such as image reconstruction [5,6,7,8,9,10,11,12], image deblurring [13,14,15], object detection [16,17], and SLAM [18]

Objectives

Methods

Results

Conclusion