Dynamic Vision Sensor Research Articles

Evolution of robust high speed optical-flow-based landing for autonomous MAVs

Automatic optimization of robotic behavior has been the long-standing goal of Evolutionary Robotics. Allowing the problem at hand to be solved by automation often leads to novel approaches and new insights. A common problem encountered with this approach is that when this optimization occurs in a simulated environment, the optimized policies are subject to the reality gap when implemented in the real world. This often results in sub-optimal behavior, if it works at all. This paper investigates the automatic optimization of neurocontrollers to perform quick but safe landing maneuvers for a quadrotor micro air vehicle using the divergence of the optical flow field of a downward looking camera. The optimized policies showed that a piece-wise linear control scheme is more effective than the simple linear scheme commonly used, something not yet considered by human designers. Additionally, we show the utility in using abstraction on the input and output of the controller as a tool to improve the robustness of the optimized policies to the reality gap by testing our policies optimized in simulation on real world vehicles. We tested the neurocontrollers using two different methods to generate and process the visual input, one using a conventional CMOS camera and one a dynamic vision sensor, both of which perform significantly differently than the simulated sensor. The use of the abstracted input resulted in near seamless transfer to the real world with the controllers showing high robustness to a clear reality gap.

Neural State Machines for Robust Learning and Control of Neuromorphic Agents

Mixed-signal analog/digital neuromorphic circuits are characterized by ultra-low power consumption, real-time processing abilities, and low-latency response times. These features make them promising for robotic applications that require fast and power-efficient computing. However, due to the device mismatch and variability present in these circuits, developing architectures that can perform complex computations in a robust and reproducible manner is quite challenging. In this paper, we present a spiking neural network architecture implemented using these neuromorphic circuits, that enables reliable control of an autonomous agent as well as robust learning and recognition of visual patterns in a noisy real-world environment. While learning is implemented with a software algorithm running with a chip-in-the-loop setup, the inference and motor control processes are implemented exclusively by the neuromorphic processor, situated on the neuromorphic agent. In addition to this processor device, the agent comprises a dynamic vision sensor which produces spikes as it interacts with the environment in real-time. We show how the robust learning and reliable control properties of the system arise out of a recently proposed neural computational primitive denoted as Neural State Machine (NSM). We describe the features of the NSMs used in this context and demonstrate the agent's real-time robust perception and action behavior with experimental results.

DAViS Camera Optical Flow

Frame-based optical flow (OF) methods struggle in the presence of large motion and occlusion due to slow frame rates. Optical flow of dynamic vision sensor (DVS) has gained attention recently as a way to overcome these shortcomings. DVS reports “events” corresponding log intensity changes exceeding a specific threshold, with an accuracy of microsecond order. The increased temporal sampling rate are indeed helpful, but the poor spatial fidelity of DVS outputs make events-based OF less reliable overall. In this work, we consider a new sensor called DAViS that combines the conventional active pixel sensor (APS) and DVS circuitries, yielding a conventional intensity image frames as well as the events. We propose a novel optical flow method designed specifically for a DAViS camera that leverages the high spatial fidelity of intensity image frames and the high temporal resolution of events generated by DVS. Hence, the proposed DAViS-OF method yields reliable motion vector estimates while overcoming the fast motion and occlusion problems. The proposed DAViS-OF method is computationally efficient and is suitable for real-time implementation.

Incremental Learning of Hand Symbols Using Event-Based Cameras

Conventional cameras create redundant output especially when the frame rate is high. Dynamic vision sensors (DVSs), on the other hand, generate asynchronous and sparse brightness change events only when an object in the field of view is in motion. Such event-based output can be processed as a 1D time sequence, or it can be converted to 2D frames that resemble conventional camera frames. Frames created, e.g., by accumulating a fixed number of events, can be used as input for conventional deep learning algorithms, thus upgrading existing computer vision pipelines through low-power, low-redundancy sensors. This paper describes a hand symbol recognition system that can quickly be trained to incrementally learn new symbols recorded with an event-based camera, without forgetting previously learned classes. By using the iCaRL incremental learning algorithm, we show that we can learn up to 16 new symbols using only 4000 samples for each symbol and achieving a final symbol accuracy of over 80%. The system achieves latency of under 0.5s and training requires 3 minutes for 5 epochs on an NVIDIA 1080TI GPU.

Fast, Flexible Closed-Loop Feedback: Tracking Movement in "Real-Millisecond-Time".

One of the principal functions of the brain is to control movement and rapidly adapt behavior to a changing external environment. Over the last decades our ability to monitor activity in the brain, manipulate it while also manipulating the environment the animal moves through, has been tackled with increasing sophistication. However, our ability to track the movement of the animal in real time has not kept pace. Here, we use a dynamic vision sensor (DVS) based event-driven neuromorphic camera system to implement real-time, low-latency tracking of a single whisker that mice can move at ∼25 Hz. The customized DVS system described here converts whisker motion into a series of events that can be used to estimate the position of the whisker and to trigger a position-based output interactively within 2 ms. This neuromorphic chip-based closed-loop system provides feedback rapidly and flexibly. With this system, it becomes possible to use the movement of whiskers or in principal, movement of any part of the body to reward, punish, in a rapidly reconfigurable way. These methods can be used to manipulate behavior, and the neural circuits that help animals adapt to changing values of a sequence of motor actions.

Parameter Optimization and Learning in a Spiking Neural Network for UAV Obstacle Avoidance Targeting Neuromorphic Processors

The Lobula giant movement detector (LGMD) is an identified neuron of the locust that detects looming objects and triggers the insect's escape responses. Understanding the neural principles and network structure that leads to these fast and robust responses can facilitate the design of efficient obstacle avoidance strategies for robotic applications. Here, we present a neuromorphic spiking neural network model of the LGMD driven by the output of a neuromorphic dynamic vision sensor (DVS), which incorporates spiking frequency adaptation and synaptic plasticity mechanisms, and which can be mapped onto existing neuromorphic processor chips. However, as the model has a wide range of parameters and the mixed-signal analog-digital circuits used to implement the model are affected by variability and noise, it is necessary to optimize the parameters to produce robust and reliable responses. Here, we propose to use differential evolution (DE) and Bayesian optimization (BO) techniques to optimize the parameter space and investigate the use of self-adaptive DE (SADE) to ameliorate the difficulties of finding appropriate input parameters for the DE technique. We quantify the performance of the methods proposed with a comprehensive comparison of different optimizers applied to the model and demonstrate the validity of the approach proposed using recordings made from a DVS sensor mounted on an unmanned aerial vehicle (UAV).

Robust Event-Based Object Tracking Combining Correlation Filter and CNN Representation.

Object tracking based on the event-based camera or dynamic vision sensor (DVS) remains a challenging task due to the noise events, rapid change of event-stream shape, chaos of complex background textures, and occlusion. To address the challenges, this paper presents a robust event-stream object tracking method based on correlation filter mechanism and convolutional neural network (CNN) representation. In the proposed method, rate coding is used to encode the event-stream object. Feature representations from hierarchical convolutional layers of a pre-trained CNN are used to represent the appearance of the rate encoded event-stream object. Results prove that the proposed method not only achieves good tracking performance in many complicated scenes with noise events, complex background textures, occlusion, and intersected trajectories, but also is robust to variable scale, variable pose, and non-rigid deformations. In addition, the correlation filter-based method has the advantage of high speed. The proposed approach will promote the potential applications of these event-based vision sensors in autonomous driving, robots and many other high-speed scenes.

Analysis of Input-Dependent Noise in Self-Timed Reset Dynamic Vision Sensor and its Impact on Data Quality

Threshold temporal contrast (TTC), as a key feature of event-based signal encoding scheme, always fluctuates due to the various noise sources in circuit implementation. This fluctuation (or noise) distorts the information that each event contains, leading to the data quality degeneration. To avoid the errors that may appear in the subsequent signal processing, the degeneration needs to be within the limit demanded by the algorithm. Therefore, modeling the relationship between noise and data quality is important for the event-based hardware application. This paper focuses on analyzing the input-dependent noise on TTC caused by dark current and charge injection, which are labeled as $\delta \Theta _{\mathrm {dc}}$ and $\delta \Theta _{\mathrm {ci}}$ respectively, in self-timed reset dynamic vision sensors (STR-DVS), and evaluates the corresponding impacts on data quality. As for the theoretical derivation part, $\delta \Theta _{\mathrm {dc}}$ is calculated using a logarithmic photocurrent-to-voltage model with dark current taken into account, and $\delta \Theta _{\mathrm {ci}}$ is figured out with a three-phase on-to-off-transient switch model applied in differencing circuit. The data quality evaluation is implemented with event clusters generated from the nonideality-contained model, and the signal-to-distortion ratio (SDR) of the signal reconstructed by a perfect nonlinear recovery algorithm is taken as the metric. The analysis and simulation results are expected to offer a reference on assessing the applicability of STR-DVS for a specific application, or system parameter settings to make the data quality satisfy the requirement while using the sensor.

Neuromorphic Vision Datasets for Pedestrian Detection, Action Recognition, and Fall Detection.

Neuromorphic Vision Datasets for Pedestrian Detection, Action Recognition, and Fall Detection.

Bandwidth Modeling of Silicon Retinas for Next Generation Visual Sensor Networks.

Silicon retinas, also known as Dynamic Vision Sensors (DVS) or event-based visual sensors, have shown great advantages in terms of low power consumption, low bandwidth, wide dynamic range and very high temporal resolution. Owing to such advantages as compared to conventional vision sensors, DVS devices are gaining more and more attention in various applications such as drone surveillance, robotics, high-speed motion photography, etc. The output of such sensors is a sequence of events rather than a series of frames as for classical cameras. Estimating the data rate of the stream of events associated with such sensors is needed for the appropriate design of transmission systems involving such sensors. In this work, we propose to consider information about the scene content and sensor speed to support such estimation, and we identify suitable metrics to quantify the complexity of the scene for this purpose. According to the results of this study, the event rate shows an exponential relationship with the metric associated with the complexity of the scene and linear relationships with the speed of the sensor. Based on these results, we propose a two-parameter model for the dependency of the event rate on scene complexity and sensor speed. The model achieves a prediction accuracy of approximately 88.4% for the outdoor environment along with the overall prediction performance of approximately 84%.

Characterization setup for event-based imagers applied to modulated light signal detection.

Event-based cameras bring new perspectives for perception systems by making them faster, smarter, and less energy-consuming. While they are spreading into many application domains, new algorithms are designed to process the data they provide, and new databases are needed to validate and train them. Simulations are an efficient way to increase databases, as they give direct access to ground truth for applications such as target detection or depth estimation, provided the simulation models used are as close as possible to the physical reality. The model should also be designed generically enough to be applicable to different kinds of event-based imagers. The characterization setup proposed in this paper aims at measuring the main characteristics of the dynamic vision sensor in each pixel under outdoor lighting conditions. A simulation model of the imager's response can be generated using the measured characteristics. These measurements are used to estimate the robustness of an algorithm to detect modulated light signals exploiting event-based data. An improvement is then provided so this algorithm can detect higher frequencies.

Spike Coding for Dynamic Vision Sensor in Intelligent Driving

Dynamic vision sensor (DVS) as a bio-inspired camera, has shown great advantages in wide dynamic range and high temporal resolution imaging in contrast to conventional frame-based cameras. Its ability to capture high speed moving objects enables fast and accurate detection which plays a significant role in the emerging intelligent driving applications. The pixels in DVS independently respond to the luminance changes with output spikes. Thus, the spike stream conveying the ${x}$ -, ${y}$ -addresses, the firing time, and the polarity (ON/OFF), is quite different from conventional video frames. How to compress this kind of new data for efficient transmission and storage remains a big challenge, especially for on-board detection, monitoring and recording. To address this challenge, this paper first analyzes the spike firing mechanism and the spatiotemporal characteristics of the spike data, then introduces a cube-based spike coding framework for DVS. In the framework, an octree-based structure is proposed to adaptively partition the spike stream into coding cubes in both spatial and temporal dimensions, then several prediction modes are designed to exploit the spatial and temporal characteristics of spikes for compression, including address-prior mode and time-prior mode. To explore more flexibility, the intercube prediction is discussed extensively involving motion estimation and motion compensation. Finally, the experimental results demonstrate that our approach achieves an impressive coding performance with the average compression ratio of 2.6536 against the raw spike data, which is much higher than the results of conventional lossless coding algorithms.

Low Latency Event-Based Filtering and Feature Extraction for Dynamic Vision Sensors in Real-Time FPGA Applications

Dynamic Vision Sensor (DVS) pixels produce an asynchronous variable-rate address-event output that represents brightness changes at the pixel. Since these sensors produce frame-free output, they are ideal for real-time dynamic vision applications with real-time latency and power system constraints. Event-based filtering algorithms have been proposed to post-process the asynchronous event output to reduce sensor noise, extract low level features, and track objects, among others. These postprocessing algorithms help to increase the performance and accuracy of further processing for tasks such as classification using spike-based learning (ie. ConvNets), stereo vision, and visually-servoed robots, etc. This paper presents an FPGA-based library of these postprocessing event-based algorithms with implementation details; specifically background activity (noise) filtering, pixel masking, object motion detection and object tracking. The latencies of these filters on the Field Programmable Gate Array (FPGA) platform are below 300ns with an average latency reduction of 188% (maximum of 570%) over the software versions running on a desktop PC CPU. This open-source event-based filter IP library for FPGA has been tested on two different platforms and scenarios using different synthesis and implementation tools for Lattice and Xilinx vendors.

Bio-Inspired Stereo Vision Calibration for Dynamic Vision Sensors

Many advances have been made in the field of computer vision. Several recent research trends have focused on mimicking human vision by using a stereo vision system. In multi-camera systems, a calibration process is usually implemented to improve the results accuracy. However, these systems generate a large amount of data to be processed; therefore, a powerful computer is required and, in many cases, this cannot be done in real time. Neuromorphic Engineering attempts to create bio-inspired systems that mimic the information processing that takes place in the human brain. This information is encoded using pulses (or spikes) and the generated systems are much simpler (in computational operations and resources), which allows them to perform similar tasks with much lower power consumption, thus these processes can be developed over specialized hardware with real-time processing. In this work, a bio-inspired stereo-vision system is presented, where a calibration mechanism for this system is implemented and evaluated using several tests. The result is a novel calibration technique for a neuromorphic stereo vision system, implemented over specialized hardware (FPGA - Field-Programmable Gate Array), which allows obtaining reduced latencies on hardware implementation for stand-alone systems, and working in real time.

Pedestrian Detection and Tracking Using a Dynamic Vision Sensor

Neuromorphic sensors such as the DynamicVision Sensor (DVS) emulate the behavior of the primary vision system. Its asynchronous behavior makes the data processing easier and faster due to the analysis is only in the active pixels. Pedestrian kinematics contains specific movement patterns feasible to be detected, like the angular movement of arms and feet. Some previous methodologies were focused on pedestrian detection based on the static shapes detection like cylinders orcircles, however, they do not take into account the kinematic behavior of the body by itself. In this paper, we presented an algorithm inspired in K-means clustering and describes the analysis of the human kinematics based on DVS in order to detect and track pedestrians in a controlled environment.

Super-resolution of spatiotemporal event-stream image

Super-resolution (SR) is a useful technology to generate a high-resolution (HR) visual output from the low-resolution (LR) visual inputs overcoming the physical limitations of the cameras. However, SR has not been applied to enhance the spatial resolution of event-stream images captured by the frame-free dynamic vision sensor (DVS). SR of event-stream image aims to recover the same statistics of events which is fundamentally different from the existing frame-based schemes. In this work, a two-stage scheme is proposed to solve the spatial SR problem of the spatiotemporal event-stream image. We use a nonhomogeneous Poisson point process to model the event sequence, and sample the events of each pixel by simulating a nonhomogeneous Poisson process according to the specified event number and rate function. Firstly, the event number of each pixel of the HR DVS image is generated by obtaining the HR event-count map (ECM) from the LR DVS recording with a sparse representation based method. The rate function over time line of the point process of each HR pixel is computed using a spatiotemporal filter on the corresponding LR neighbor pixels. Secondly, the event sequence of each new pixel is obtained with a thinning based event sampling algorithm. A metric is proposed to assess the event-stream SR quality. The effectiveness of the proposed method is demonstrated through obtaining HR event-stream images from a series of DVS recordings. This work enables many potential.

Hough Transform Implementation For Event-Based Systems: Concepts and Challenges.

Hough transform (HT) is one of the most well-known techniques in computer vision that has been the basis of many practical image processing algorithms. HT however is designed to work for frame-based systems such as conventional digital cameras. Recently, event-based systems such as Dynamic Vision Sensor (DVS) cameras, has become popular among researchers. Event-based cameras have a significantly high temporal resolution (1 μs), but each pixel can only detect change and not color. As such, the conventional image processing algorithms cannot be readily applied to event-based output streams. Therefore, it is necessary to adapt the conventional image processing algorithms for event-based cameras. This paper provides a systematic explanation, starting from extending conventional HT to 3D HT, adaptation to event-based systems, and the implementation of the 3D HT using Spiking Neural Networks (SNNs). Using SNN enables the proposed solution to be easily realized on hardware using FPGA, without requiring CPU or additional memory. In addition, we also discuss techniques for optimal SNN-based implementation using efficient number of neurons for the required accuracy and resolution along each dimension, without increasing the overall computational complexity. We hope that this will help to reduce the gap between event-based and frame-based systems.

On Practical Issues for Stochastic STDP Hardware With 1-bit Synaptic Weights.

In computational neuroscience, synaptic plasticity learning rules are typically studied using the full 64-bit floating point precision computers provide. However, for dedicated hardware implementations, the precision used not only penalizes directly the required memory resources, but also the computing, communication, and energy resources. When it comes to hardware engineering, a key question is always to find the minimum number of necessary bits to keep the neurocomputational system working satisfactorily. Here we present some techniques and results obtained when limiting synaptic weights to 1-bit precision, applied to a Spike-Timing-Dependent-Plasticity (STDP) learning rule in Spiking Neural Networks (SNN). We first illustrate the 1-bit synapses STDP operation by replicating a classical biological experiment on visual orientation tuning, using a simple four neuron setup. After this, we apply 1-bit STDP learning to the hidden feature extraction layer of a 2-layer system, where for the second (and output) layer we use already reported SNN classifiers. The systems are tested on two spiking datasets: a Dynamic Vision Sensor (DVS) recorded poker card symbols dataset and a Poisson-distributed spike representation MNIST dataset version. Tests are performed using the in-house MegaSim event-driven behavioral simulator and by implementing the systems on FPGA (Field Programmable Gate Array) hardware.

Comment on “Temperature and Parasitic Photocurrent Effects in Dynamic Vision Sensors”

In the title paper, the temperature dependence of event threshold and leak activity frequency of dynamic vision sensor are researched, and a few new metrics are presented to describe the features of the leak event. A large amount of experimental data is shown to verify the theories. We comment on this paper, since Nozaki and Delbruck make some mistakes on formula derivation and the experimental data calculation. The errors are pointed out with detailed explanations, and the revised formulas and calculation results, together with some corresponding analyses, are provided.

Authors’ Reply to Comment on “Temperature and Parasitic Photocurrent Effects in Dynamic Vision Sensors”

We thank the reviewers for their careful analysis of [1], especially for spotting two errors in the formulas for inferring temporal contrast effects of leak and parasitic photocurrent. The revised results increase the values of the inferred parasitic leak currents by a factor of about 11× .