Dynamic Vision Sensor Research Articles

Target detection and obstacle avoidance for UAV based on dynamic vision sensor

Aiming at the problem of UAV sensing dynamic targets and avoiding high-speed dynamic obstacles in dynamic environments, a target detection and obstacle avoidance algorithm based on dynamic vision sensor is designed. A filtering method and motion compensation algorithm are designed to filter out background noise and hot spot noise in the event stream as well as redundant events caused by the camera's own motion. A dynamic target fusion detection algorithm that fuses event images and RGB images is designed to ensure the reliability of detection. The target motion trajectory is estimated based on the detection results, and the speed obstacle method is improved to avoid dynamic obstacles by combining the obstacle motion characteristics and UAV dynamic constraints. A large number of simulation tests, handheld tests and flight tests verify the feasibility of the proposed algorithm.

BrainQN: Enhancing the Robustness of Deep Reinforcement Learning with Spiking Neural Networks

As the third‐generation network succeeding artificial neural networks (ANNs), spiking neural networks (SNNs) offer high robustness and low energy consumption. Inspired by biological systems, the limitations of low robustness and high‐power consumption in deep reinforcement learning (DRL) are addressed by introducing SNNs. The Brain Q‐network (BrainQN) is proposed, which replaces the neurons in the classic Deep Q‐learning (DQN) algorithm with SNN neurons. BrainQN is trained using surrogate gradient learning (SGL) and ANN‐to‐SNN conversion methods. Robustness tests with input noise reveal BrainQN's superior performance, achieving an 82.14% increase in rewards under low noise and 71.74% under high noise compared to DQN. These findings highlight BrainQN's robustness and superior performance in noisy environments, supporting its application in complex scenarios. SGL‐trained BrainQN is more robust than ANN‐to‐SNN conversion under high noise. The differences in network output correlations between noisy and original inputs, along with training algorithm distinctions, explain this phenomenon. BrainQN successfully transitioned from a simulated Pong environment to a ball‐catching robot with dynamic vision sensors (DVS). On the neuromorphic chip PAICORE, it shows significant advantages in latency and power consumption compared to Jetson Xavier NX.

An improved pixel circuit with low noise event rate and enhanced bright‐light sensitivity for dynamic vision sensor

AbstractDynamic vision sensor (DVS) imaging quality is significantly affected by pixel noise and temporal contrast (TC), which is inversely proportional to sensitivity. To reduce the noise event rate and improve sensitivity in bright‐light conditions in the DVS pixel circuit, this paper proposes improvements to the conventional DVS pixel circuit. The proposed DVS pixel circuit adopts stacked medium‐threshold transistors instead of a single high‐threshold transistor in the photoreceptor and introduces a threshold switching circuit. Compared with the conventional DVS pixel circuit, this design increases event threshold normalized by root mean square (RMS) noise voltage, reducing the dim‐light noise bandwidth. Additionally, it achieves higher sensitivity in bright‐light conditions compared with dim‐light conditions. The proposed DVS pixel circuit is implemented in a 110‐nm complementary metal‐oxide semiconductor (CMOS) process. Post‐simulation results show that, for photocurrents between 5 fA and 100 pA, the proposed DVS pixel circuit achieves a 35 Hz peak event rate at 15% TC, which is reduced to 3.1% of the conventional structure. For photocurrents exceeding 30 pA, the proposed structure can switch TC from 15% to 5%, maintaining a noise event rate below 0.1 Hz.

Motion-error-free calibration of event camera systems using a flashing target

Event cameras, inspired by biological vision, offer high dynamic range, excellent temporal resolution, and minimal data redundancy. Precise calibration of event camera systems is essential for applications such as 3D vision. The cessation of extra gray frame production in popular models like the dynamic vision sensor (DVS) poses significant challenges to achieving high-accuracy calibration. Traditional calibration methods, which rely on motion to trigger events, are prone to movement-related errors. This paper introduces a motion-error-free calibration method for event cameras using a flashing target produced by a standard electronic display that elicits high-fidelity events. We propose an improved events-accumulator to reconstruct gray images with distinct calibration features and develop an optimization method that adjusts camera parameters and control point positions simultaneously, enhancing the calibration accuracy of event camera systems. Experimental results demonstrated higher accuracy compared to the traditional motion-based calibration method (reprojection error: 0.03 vs. 0.96 pixels). The 3D reconstruction error remained around 0.15 mm, significantly improving over the motion-based method’s 8.00 mm. Additionally, the method’s adaptability for hybrid calibration in event-based stereovision systems was verified (e.g., with frame cameras or projectors).

A dual threshold adaptation method to dynamic vision sensor pixel

This paper introduces a method for the dynamic vision sensor (DVS) pixel that achieves dual-threshold adaptation based on the interval between adjacent event spikes. The Time-to-Amplitude Converter (TAC) measures the adjacent spike interval per pixel. Based on the output of the TAC and two predefined time limits, the threshold selection circuit automatically chooses between the low threshold ΘL and the high threshold ΘH. The proposed pixel pitch is 33.2μm×34.2μm with 8.1% fill factor in a 0.11μm 1P4M CMOS process. Compared to the traditional pixel with fixed threshold temporal contrast Θ = 15%, the behavior model indicates that the proposed structure maintains almost equivalent detection ability while reducing event output by 30% when ΘH = 45% and ΘL = 15%. Compared to the traditional pixel with fixed Θ = 45%, the behavior model indicates that the average gradient of the proposed structure is improved by at least 3 times with ΘH = 45% and ΘL = 15%. This enables the detection of low temporal contrast signals while minimizing the quantity of output events.

Event-based asynchronous HDR imaging by temporal incident light modulation

Dynamic range (DR) is a pivotal characteristic of imaging systems. Current frame-based cameras struggle to achieve high dynamic range imaging due to the conflict between globally uniform exposure and spatially variant scene illumination. In this paper, we propose AsynHDR, a pixel-asynchronous HDR imaging system, based on key insights into the challenges in HDR imaging and the unique event-generating mechanism of dynamic vision sensors (DVS). Our proposed AsynHDR system integrates the DVS with a set of LCD panels. The LCD panels modulate the irradiance incident upon the DVS by altering their transparency, thereby triggering the pixel-independent event streams. The HDR image is subsequently decoded from the event streams through our temporal-weighted algorithm. Experiments under the standard test platform and several challenging scenes have verified the feasibility of the system in HDR imaging tasks.

E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning

The bio-inspired event cameras or dynamic vision sensors are capable of asynchronously capturing per-pixel brightness changes (called event-streams) in high temporal resolution and high dynamic range. However, the non-structural spatial-temporal event-streams make it challenging for providing intuitive visualization with rich semantic information for human vision. It calls for events-to-video (E2V) solutions which take event-streams as input and generate high quality video frames for intuitive visualization. However, current solutions are predominantly data-driven without considering the prior knowledge of the underlying statistics relating event-streams and video frames. It highly relies on the non-linearity and generalization capability of the deep neural networks, thus, is struggling on reconstructing detailed textures when the scenes are complex. In this work, we propose E2HQV, a novel E2V paradigm designed to produce high-quality video frames from events. This approach leverages a model-aided deep learning framework, underpinned by a theory-inspired E2V model, which is meticulously derived from the fundamental imaging principles of event cameras. To deal with the issue of state-reset in the recurrent components of E2HQV, we also design a temporal shift embedding module to further improve the quality of the video frames. Comprehensive evaluations on the real world event camera datasets validate our approach, with E2HQV, notably outperforming state-of-the-art approaches, e.g., surpassing the second best by over 40% for some evaluation metrics.

Memristor-CMOS Hybrid Circuits Implementing Event-Driven Neural Networks for Dynamic Vision Sensor Camera.

For processing streaming events from a Dynamic Vision Sensor camera, two types of neural networks can be considered. One are spiking neural networks, where simple spike-based computation is suitable for low-power consumption, but the discontinuity in spikes can make the training complicated in terms of hardware. The other one are digital Complementary Metal Oxide Semiconductor (CMOS)-based neural networks that can be trained directly using the normal backpropagation algorithm. However, the hardware and energy overhead can be significantly large, because all streaming events must be accumulated and converted into histogram data, which requires a large amount of memory such as SRAM. In this paper, to combine the spike-based operation with the normal backpropagation algorithm, memristor-CMOS hybrid circuits are proposed for implementing event-driven neural networks in hardware. The proposed hybrid circuits are composed of input neurons, synaptic crossbars, hidden/output neurons, and a neural network's controller. Firstly, the input neurons perform preprocessing for the DVS camera's events. The events are converted to histogram data using very simple memristor-based latches in the input neurons. After preprocessing the events, the converted histogram data are delivered to an ANN implemented using synaptic memristor crossbars. The memristor crossbars can perform low-power Multiply-Accumulate (MAC) calculations according to the memristor's current-voltage relationship. The hidden and output neurons can convert the crossbar's column currents to the output voltages according to the Rectified Linear Unit (ReLU) activation function. The neural network's controller adjusts the MAC calculation frequency according to the workload of the event computation. Moreover, the controller can disable the MAC calculation clock automatically to minimize unnecessary power consumption. The proposed hybrid circuits have been verified by circuit simulation for several event-based datasets such as POKER-DVS and MNIST-DVS. The circuit simulation results indicate that the neural network's performance proposed in this paper is degraded by as low as 0.5% while saving as much as 79% in power consumption for POKER-DVS. The recognition rate of the proposed scheme is lower by 0.75% compared to the conventional one, for the MNIST-DVS dataset. In spite of this little loss, the power consumption can be reduced by as much as 75% for the proposed scheme.

An event-oriented diffusion-refinement method for sparse events completion

Event cameras or dynamic vision sensors (DVS) record asynchronous response to brightness changes instead of conventional intensity frames, and feature ultra-high sensitivity at low bandwidth. The new mechanism demonstrates great advantages in challenging scenarios with fast motion and large dynamic range. However, the recorded events might be highly sparse due to either limited hardware bandwidth or extreme photon starvation in harsh environments. To unlock the full potential of event cameras, we propose an inventive event sequence completion approach conforming to the unique characteristics of event data in both the processing stage and the output form. Specifically, we treat event streams as 3D event clouds in the spatiotemporal domain, develop a diffusion-based generative model to generate dense clouds in a coarse-to-fine manner, and recover exact timestamps to maintain the temporal resolution of raw data successfully. To validate the effectiveness of our method comprehensively, we perform extensive experiments on three widely used public datasets with different spatial resolutions, and additionally collect a novel event dataset covering diverse scenarios with highly dynamic motions and under harsh illumination. Besides generating high-quality dense events, our method can benefit downstream applications such as object classification and intensity frame reconstruction.

SGLFormer: Spiking Global-Local-Fusion Transformer with high performance.

Spiking Neural Networks (SNNs), inspired by brain science, offer low energy consumption and high biological plausibility with their event-driven nature. However, the current SNNs are still suffering from insufficient performance. Recognizing the brain's adeptness at information processing for various scenarios with complex neuronal connections within and across regions, as well as specialized neuronal architectures for specific functions, we propose a Spiking Global-Local-Fusion Transformer (SGLFormer), that significantly improves the performance of SNNs. This novel architecture enables efficient information processing on both global and local scales, by integrating transformer and convolution structures in SNNs. In addition, we uncover the problem of inaccurate gradient backpropagation caused by Maxpooling in SNNs and address it by developing a new Maxpooling module. Furthermore, we adopt spatio-temporal block (STB) in the classification head instead of global average pooling, facilitating the aggregation of spatial and temporal features. SGLFormer demonstrates its superior performance on static datasets such as CIFAR10/CIFAR100, and ImageNet, as well as dynamic vision sensor (DVS) datasets including CIFAR10-DVS and DVS128-Gesture. Notably, on ImageNet, SGLFormer achieves a top-1 accuracy of 83.73% with 64 M parameters, outperforming the current SOTA directly trained SNNs by a margin of 6.66%. With its high performance, SGLFormer can support more computer vision tasks in the future. The codes for this study can be found in https://github.com/ZhangHanN1/SGLFormer.

Measuring diameters and velocities of artificial raindrops with a neuromorphic event camera

Abstract. Hydrometers that measure size and velocity distributions of precipitation are needed for research and corrections of rainfall estimates from weather radars and microwave links. Existing optical disdrometers measure droplet size distributions, but underestimate small raindrops and are impractical for widespread always-on IoT deployment. We study the feasibility of measuring droplet size and velocity using a neuromorphic event camera. These dynamic vision sensors asynchronously output a sparse stream of pixel brightness changes. Droplets falling through the plane of focus of a steeply down-looking camera create events generated by the motion of the droplet across the field of view. Droplet size and speed are inferred from the hourglass-shaped stream of events. Using an improved hard disk arm actuator to reliably generate artificial raindrops with a range of small sizes, our experiments show maximum errors of 7 % (mean absolute percentage error) for droplet sizes from 0.3 to 2.5 mm and speeds from 1.3 to 8.0 m s−1. Measurements with the same setup from a commercial PARSIVEL disdrometer show similar results. Both devices slightly overestimate the small droplet volume with a volume overestimation of 25 % from the event camera measurements and 50 % from the PARSIVEL instrument. Each droplet requires processing of 5000 to 50 000 brightness change events, potentially enabling low-power always-on disdrometers that consume power proportional to the rainfall rate. Data and code are available at the paper website https://sites.google.com/view/dvs-disdrometer/home (Micev et al., 2023).

Global shutter CMOS vision sensors and event cameras for on‐chip dynamic information

AbstractThe on‐chip extraction of dynamic information from a scene can be addressed with either frame‐based CMOS vision, also called smart image sensors, or dynamic vision sensors, also known as event cameras. When implemented with a pinned photodiode (PPD) as a 4‐transistor active pixel sensor (4T‐APS), the former brings about the benefits of low temporal noise and dark current but without high dynamic range (HDR). The latter comes with the benefits of HDR and a fast event detection rate with low power consumption. The drawback is the background activity noise, which leads to additional hardware or algorithms to keep it low. This paperanalyses the mismatch and noise of a global shutter 4T‐APS implementation with local HDR through an overflow capacitor and correlated double sampling (CDS) to provide low noise events through frame differencing. The aim is to narrow the gap with dynamic vision sensors in terms of event rate and dynamic range. We show that our solution would be competitive with event cameras in scenarios with slow moving objects and a relatively wide dynamic range (85 dB).

E-MLB: Multilevel Benchmark for Event-Based Camera Denoising

Event cameras, such as dynamic vision sensors (DVS), are biologically inspired vision sensors that have advanced over conventional cameras in high dynamic range, low latency and low power consumption, showing great application potential in many fields. Event cameras are more sensitive to junction leakage current and photocurrent as they output differential signals, losing the smoothing function of the integral imaging process in the RGB camera. The logarithmic conversion further amplifies noise, especially in low-contrast conditions. Recently, researchers proposed a series of datasets and evaluation metrics but limitations remain: 1) the existing datasets are small in scale and insufficient in noise diversity, which cannot reflect the authentic working environments of event cameras; and 2) the existing denoising evaluation metrics are mostly referenced evaluation metrics, relying on APS information or manual annotation. To address the above issues, we construct a large-scale event denoising dataset (multilevel benchmark for event denoising, E-MLB) for the first time, which consists of 100 scenes, each with four noise levels, that is 12 times larger than the largest existing denoising dataset. We also propose the first nonreference event denoising metric, the event structural ratio (ESR), which measures the structural intensity of given events. ESR is inspired by the contrast metric, but is independent of the number of events and projection direction. Based on the proposed benchmark and ESR, we evaluate the most representative denoising algorithms, including classic and SOTA, and provide denoising baselines under various scenes and noise levels. The corresponding results and codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/KugaMaxx/cuke-emlb</uri>

A Direct Time-of-Flight Image Sensor With In-Pixel Surface Detection and Dynamic Vision

3D flash LIDAR is an alternative to the traditional scanning LIDAR systems, promising precise depth imaging in a compact form factor, and free of moving parts, for applications such as self-driving cars, robotics and augmented reality (AR). Typically implemented using single-photon, direct time-of-flight (dToF) receivers in image sensor format, the operation of the devices can be hindered by the large number of photon events needing to be processed and compressed in outdoor scenarios, limiting frame rates and scalability to larger arrays. We here present a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$64 \times 32$</tex-math></inline-formula> pixel ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$256 \times 128$</tex-math></inline-formula> SPAD) dToF imager that overcomes these limitations by using pixels with embedded histogramming, which lock onto and track the return signal. This reduces the size of output data frames considerably, enabling maximum frame rates in the 10 kFPS range or 100 kFPS for direct depth readings. The sensor offers selective readout of pixels detecting surfaces, or those sensing motion, akin to dynamic vision sensors that report changes in light intensity, leading to reduced power consumption and off-chip processing requirements. We demonstrate the application of the sensor in mid-range LIDAR.

NeuroZoom: Denoising and Super Resolving Neuromorphic Events and Spikes.

Neuromorphic cameras are emerging imaging technology that has advantages over conventional imaging sensors in several aspects including dynamic range, sensing latency, and power consumption. However, the signal-to-noise level and the spatial resolution still fall behind the state of conventional imaging sensors. In this paper, we address the denoising and super-resolution problem for modern neuromorphic cameras. We employ 3D U-Net as the backbone neural architecture for such a task. The networks are trained and tested on two types of neuromorphic cameras: a dynamic vision sensor and a spike camera. Their pixels generate signals asynchronously, the former is based on perceived light changes and the latter is based on accumulated light intensity. To collect the datasets for training such networks, we design a display-camera system to record high frame-rate videos at multiple resolutions, providing supervision for denoising and super-resolution. The networks are trained in a noise-to-noise fashion, where the two ends of the network are unfiltered noisy data. The output of the networks has been tested for downstream applications including event-based visual object tracking and image reconstruction. Experimental results demonstrate the effectiveness of improving the quality of neuromorphic events and spikes, and the corresponding improvement to downstream applications with state-of-the-art performance.

Two-stage cross-fusion network for stereo event-based depth estimation

Drawing inspiration from biological retina, event camera uses dynamic vision sensor to asynchronously capture pixel intensity changes. Event camera provides a series of advantages, including high dynamic range, high temporal resolution, lack of motion blur, and low power consumption, which make it suitable candidate for depth estimation, specially in challenging scenarios with fast motion and low-lighting conditions. However, due to the sparse nature of event data, it is troublesome to estimate dense disparity map from them. Although it is plausible to estimate disparity map from edges of a structure from where most events are captured, estimating dense disparity is challenging in the areas where events rarely occur. In this study, we develop a stereo matching network for dense disparity estimation, which effectively fuses reconstructed image features with event to estimate disparity. To this end, we propose a two-stage cross-fusion network (TCFNet) to estimate disparity by fusing event and image features with two cascaded fusion mechanisms. First, we propose cross-SPADE fusion, based on the spatially-adaptive denormalization mechanism (SPADE), which modulates event features with image features, and image features with event features. In the second stage, the modulated features are further fused using the cross-gating mechanism, which uses the multi-axis gated MLP (gMLP). Experiments on both real and synthetic datasets indicate that the proposed two-stage cross-fusion network outperforms the state-of-the-art event-based stereo matching networks empirically and visually.

Illumination-Based Color Reconstruction for the Dynamic Vision Sensor.

This work demonstrates a novel, state-of-the-art method to reconstruct colored images via the dynamic vision sensor (DVS). The DVS is an image sensor that indicates only a binary change in brightness, with no information about the captured wavelength (color) or intensity level. However, the reconstruction of the scene's color could be essential for many tasks in computer vision and DVS. We present a novel method for reconstructing a full spatial resolution, colored image utilizing the DVS and an active colored light source. We analyze the DVS response and present two reconstruction algorithms: linear-based and convolutional-neural-network-based. Our two presented methods reconstruct the colored image with high quality, and they do not suffer from any spatial resolution degradation as other methods. In addition, we demonstrate the robustness of our algorithm to changes in environmental conditions, such as illumination and distance. Finally, compared with previous works, we show how we reach the state-of-the-art results. We share our code on GitHub.

Directly-trained Spiking Neural Networks for Deep Reinforcement Learning: Energy efficient implementation of event-based obstacle avoidance on a neuromorphic accelerator

Spiking Neural Networks (SNN) promise extremely low-power and low-latency inference on neuromorphic hardware. Recent studies demonstrate the competitive performance of SNNs compared with Artificial Neural Networks (ANN) in conventional classification tasks.In this work, we present an energy-efficient implementation of a Reinforcement Learning (RL) algorithm using SNNs to solve an obstacle avoidance task performed by an Unmanned Aerial Vehicle (UAV), taking a Dynamic Vision Sensor (DVS) as event-based input. We train the SNN directly, improving upon state-of-art implementations based on hybrid (not directly trained) SNNs. For this purpose, we devise an adaptation of the Spatio-Temporal Backpropagation algorithm (STBP) for RL. We then compare the SNN with a state-of-art Convolutional Neural Network (CNN) designed to solve the same task. To this aim, we train both networks by exploiting a photorealistic training pipeline based on AirSim. To achieve a realistic latency and throughput assessment for embedded deployment, we designed and trained three different embedded SNN versions to be executed on state-of-art neuromorphic hardware, targeting state-of-the-art.We compared SNN and CNN in terms of obstacle avoidance performance showing that the SNN algorithm achieves better results than the CNN with a factor of 6× less energy. We also characterize the different SNN hardware implementations in terms of energy and spiking activity.

Denoising for Dynamic Vision Sensor Based on Augmented Spatiotemporal Correlation

To effectively denoise the output events of Dynamic Vision Sensor (DVS) in real-time, a denoising method based on augmented spatiotemporal correlation is proposed. In addition, this paper also presents a novel method to evaluate the performance of event denoising. By classifying the events and noise based on their spatiotemporal properties, we use the dynamic response characteristics of DVS under different cases to augment the spatiotemporal correlation, and construct two event filters to process different kinds of events. To benchmark event denoising, we exploit the low-latency of DVS to temporally synchronize raw event stream through motion compensation and obtain the ideal response probability of DVS to quantify the plausibility of events and measure the denoising accuracy. Experiments in various scenes validate that the denoising method is effective at reducing noise with high accuracy and little computational cost, which significantly outperforms the state-of-the-art methods. Moreover, the evaluation method can quantify the denoising accuracy and benchmark different denoising methods objectively and comprehensively.

Neuromorphic electronics for robotic perception, navigation and control: A survey

Neuromorphic electronics have great potential in the emulation of the sensory, cognitive, self-learning, and actuating functions of robots. While typically implemented in rigid silicon, emerging technologies in organic and flexible electronic materials have also led to tremendous advances in the development of neuromorphic perception systems. However, a comprehensive review of the contribution/role of organic neuromorphic electronics for robotic applications is still missing. This review presents advancements in silicon-based and organic neuromorphic electronics for intelligent robot development, focusing on perception, navigation, and learning-based control. Organic synaptic devices, along with dynamic vision sensors, enable diverse forms of sensory-enabled computational perception, offering tunability, stability, low power consumption, and conformal substrates. Integration of simultaneous localization and mapping techniques and path planning algorithms empowers robots to efficiently navigate, build accurate maps, and make informed decisions. Different learning algorithms and their hardware implementations in neuromorphic robotic control are explored, enabling robots to learn and adapt to dynamic environments. The review highlights the potential of neuromorphic electronics for sensing, thinking, and acting in advanced robotic systems. Organic, inorganic, and hybrid materials are discussed for implementing perception, navigation, and control in robots. Future research directions in the field are outlined. Leveraging various neuromorphic electronics unlocks the full potential of intelligent robotic systems for diverse applications.