Dynamic Vision Sensor Research Articles

Denoising for Dynamic Vision Sensor Based on Augmented Spatiotemporal Correlation

To effectively denoise the output events of Dynamic Vision Sensor (DVS) in real-time, a denoising method based on augmented spatiotemporal correlation is proposed. In addition, this paper also presents a novel method to evaluate the performance of event denoising. By classifying the events and noise based on their spatiotemporal properties, we use the dynamic response characteristics of DVS under different cases to augment the spatiotemporal correlation, and construct two event filters to process different kinds of events. To benchmark event denoising, we exploit the low-latency of DVS to temporally synchronize raw event stream through motion compensation and obtain the ideal response probability of DVS to quantify the plausibility of events and measure the denoising accuracy. Experiments in various scenes validate that the denoising method is effective at reducing noise with high accuracy and little computational cost, which significantly outperforms the state-of-the-art methods. Moreover, the evaluation method can quantify the denoising accuracy and benchmark different denoising methods objectively and comprehensively.

Neuromorphic electronics for robotic perception, navigation and control: A survey

Neuromorphic electronics have great potential in the emulation of the sensory, cognitive, self-learning, and actuating functions of robots. While typically implemented in rigid silicon, emerging technologies in organic and flexible electronic materials have also led to tremendous advances in the development of neuromorphic perception systems. However, a comprehensive review of the contribution/role of organic neuromorphic electronics for robotic applications is still missing. This review presents advancements in silicon-based and organic neuromorphic electronics for intelligent robot development, focusing on perception, navigation, and learning-based control. Organic synaptic devices, along with dynamic vision sensors, enable diverse forms of sensory-enabled computational perception, offering tunability, stability, low power consumption, and conformal substrates. Integration of simultaneous localization and mapping techniques and path planning algorithms empowers robots to efficiently navigate, build accurate maps, and make informed decisions. Different learning algorithms and their hardware implementations in neuromorphic robotic control are explored, enabling robots to learn and adapt to dynamic environments. The review highlights the potential of neuromorphic electronics for sensing, thinking, and acting in advanced robotic systems. Organic, inorganic, and hybrid materials are discussed for implementing perception, navigation, and control in robots. Future research directions in the field are outlined. Leveraging various neuromorphic electronics unlocks the full potential of intelligent robotic systems for diverse applications.

7 [formula omitted]J/inference end-to-end gesture recognition from dynamic vision sensor data using ternarized hybrid convolutional neural networks

Dynamic vision sensor (DVS) cameras enable energy-activity proportional visual sensing by only propagating events produced by changes in the observed scene. Furthermore, by generating these events asynchronously, they offer μs-scale latency while eliminating the redundant data transmission inherent to classical, frame-based cameras. However, the potential of DVS to improve the energy efficiency of IoT sensor nodes can only be fully realized with efficient and flexible systems that tightly integrate sensing, processing, and actuation capabilities. In this paper, we propose a complete end-to-end pipeline for DVS event data classification implemented on the Kraken parallel ultra-low power (PULP) system-on-chip and apply it to gesture recognition. A dedicated on-chip peripheral interface for DVS cameras aggregates the received events into ternary event frames. We process these video frames with a fully ternarized two-stage temporal convolutional network (TCN). The neural network can be executed either on Kraken’s PULP cluster of general-purpose RISC-V cores or on CUTIE, the on-chip ternary neural network accelerator. We perform extensive ablations on network structure, training, and data generation parameters. We achieve a validation accuracy of 97.7% on the DVS128 11-class gesture dataset, a new record for embedded implementations. With in-silicon power and energy measurements, we demonstrate a classification energy of 7 μJ at a latency of 0.9ms when running the TCN on CUTIE, a reduction of inference energy by 67× when compared to the state of the art in embedded gesture recognition. The processing system consumes as little as 4.7mW in continuous inference, enabling always-on gesture recognition and closing the gap between the efficiency potential of DVS cameras and application scenarios.

BIDL: a brain-inspired deep learning framework for spatiotemporal processing.

Brain-inspired deep spiking neural network (DSNN) which emulates the function of the biological brain provides an effective approach for event-stream spatiotemporal perception (STP), especially for dynamic vision sensor (DVS) signals. However, there is a lack of generalized learning frameworks that can handle various spatiotemporal modalities beyond event-stream, such as video clips and 3D imaging data. To provide a unified design flow for generalized spatiotemporal processing (STP) and to investigate the capability of lightweight STP processing via brain-inspired neural dynamics, this study introduces a training platform called brain-inspired deep learning (BIDL). This framework constructs deep neural networks, which leverage neural dynamics for processing temporal information and ensures high-accuracy spatial processing via artificial neural network layers. We conducted experiments involving various types of data, including video information processing, DVS information processing, 3D medical imaging classification, and natural language processing. These experiments demonstrate the efficiency of the proposed method. Moreover, as a research framework for researchers in the fields of neuroscience and machine learning, BIDL facilitates the exploration of different neural models and enables global-local co-learning. For easily fitting to neuromorphic chips and GPUs, the framework incorporates several optimizations, including iteration representation, state-aware computational graph, and built-in neural functions. This study presents a user-friendly and efficient DSNN builder for lightweight STP applications and has the potential to drive future advancements in bio-inspired research.

Denoising Method Based on Salient Region Recognition for the Spatiotemporal Event Stream.

Event cameras are the emerging bio-mimetic sensors with microsecond-level responsiveness in recent years, also known as dynamic vision sensors. Due to the inherent sensitivity of event camera hardware to light sources and interference from various external factors, various types of noises are inevitably present in the camera's output results. This noise can degrade the camera's perception of events and the performance of algorithms for processing event streams. Moreover, since the output of event cameras is in the form of address-event representation, efficient denoising methods for traditional frame images are no longer applicable in this case. Most existing denoising methods for event cameras target background activity noise and sometimes remove real events as noise. Furthermore, these methods are ineffective in handling noise generated by high-frequency flickering light sources and changes in diffused light reflection. To address these issues, we propose an event stream denoising method based on salient region recognition in this paper. This method can effectively remove conventional background activity noise as well as irregular noise caused by diffuse reflection and flickering light source changes without significantly losing real events. Additionally, we introduce an evaluation metric that can be used to assess the noise removal efficacy and the preservation of real events for various denoising methods.

EV-Perturb: event-stream perturbation for privacy-preserving classification with dynamic vision sensors

EV-Perturb: event-stream perturbation for privacy-preserving classification with dynamic vision sensors

Vertically integrated spiking cone photoreceptor arrays for color perception

The cone photoreceptors in our eyes selectively transduce the natural light into spiking representations, which endows the brain with high energy-efficiency color vision. However, the cone-like device with color-selectivity and spike-encoding capability remains challenging. Here, we propose a metal oxide-based vertically integrated spiking cone photoreceptor array, which can directly transduce persistent lights into spike trains at a certain rate according to the input wavelengths. Such spiking cone photoreceptors have an ultralow power consumption of less than 400 picowatts per spike in visible light, which is very close to biological cones. In this work, lights with three wavelengths were exploited as pseudo-three-primary colors to form ‘colorful’ images for recognition tasks, and the device with the ability to discriminate mixed colors shows better accuracy. Our results would enable hardware spiking neural networks with biologically plausible visual perception and provide great potential for the development of dynamic vision sensors.

The Hardware Impact of Quantization and Pruning for Weights in Spiking Neural Networks

Energy efficient implementations and deployments of Spiking neural networks (SNNs) have been of great interest due to the possibility of developing artificial systems that can achieve the computational powers and energy efficiency of the biological brain. Efficient implementations of SNNs on modern digital hardware are also inspired by advances in machine learning and deep neural networks (DNNs). Two techniques widely employed in the efficient deployment of DNNs – the quantization and pruning of parameters, can both compress the model size, reduce memory footprints, and facilitate low-latency execution. The interaction between quantization and pruning and how they might impact model performance on SNN accelerators is currently unknown. We study various combinations of pruning and quantization in isolation, cumulatively, and simultaneously (jointly) to a state-of-the-art SNN targeting gesture recognition for dynamic vision sensor cameras (DVS). We show that this state-of-the-art model is amenable to aggressive parameter quantization, not suffering from any loss in accuracy down to ternary weights. However, pruning only maintains iso-accuracy up to 80% sparsity, which results in 45% more energy than the best quantization on our architectural model. Applying both pruning and quantization can result in an accuracy loss to offer a favourable trade-off on the energy-accuracy Pareto-frontier for the given hardware configuration.1

이벤트 센서를 활용한 운전 중 장비 상태 감시 가능성 고찰

Vibration-based condition monitoring technology is regarded as one of the most effective maintenance technologies. Many studies aimed at developing a vision sensor-based condition monitoring system using image processing technologies are being conducted. Event sensors such as dynamic vision sensor have a different operating principle and have many advantages compared to the traditional frame-based sensors; however, for event sensor-based vibration measurement and analysis, the formalized methodology does not exist as an image processing technology. In this study, optical flow and template matching methods were used as tracking methods for event sensor-based condition monitoring. The vibration measurement performance of the representative methods was examined through experiments using a simulator with vertical and horizontal movements. Furthermore, the possibility of extracting changes in the vibration signals according to the state of simulator was investigated.

Event-based micro vibration measurement using phase correlation template matching with event filter optimization

Condition monitoring of mechanical systems based on vibration is one of the most effective and useful techniques for predictive maintenance. With advancements in image processing and artificial intelligence technologies, vision camera-based condition monitoring has come into the spotlight. An event camera, also called a dynamic vision sensor, operates on a principle different from that of traditional frame-based cameras. However, there is still no standardized method for event camera-based vibration measurement. This study proposes an event filter-based phase correlation template match (EF-PCTM) method, including its optimal design procedure, to measure micro-vibrations using an event camera. In this study, event filter is designed using an infinite impulse response filter and genetic algorithm, and a cost function is defined to improve the performance of EF-PCTM. The performance of the proposed method was verified by two experiments using a rotor system simulator: i) extraction of micro-vibration and ii) fault condition monitoring. The proposed image processing procedure using EF-PCTM makes it possible to measure micro-vibrations, which had been challenging to measure in the past, and calculate the operational deflection shape.

Visualization and Object Detection Based on Event Information.

A dynamic vision sensor is an optical sensor that focuses on dynamic changes and outputs event information containing only position, time, and polarity. It has the advantages of high temporal resolution, high dynamic range, low data volume, and low power consumption. However, a single event can only indicate that the increase or decrease in light exceeds the threshold at a certain pixel position and a certain moment. In order to further study the ability and characteristics of event information to represent targets, this paper proposes an event information visualization method with adaptive temporal resolution. Compared with methods with constant time intervals and a constant number of events, it can better convert event information into pseudo-frame images. Additionally, in order to explore whether the pseudo-frame image can efficiently complete the task of target detection according to its characteristics, this paper designs a target detection network named YOLOE. Compared with other algorithms, it has a more balanced detection effect. By constructing a dataset and conducting experimental verification, the detection accuracy of the image obtained by the event information visualization method with adaptive temporal resolution was 5.11% and 4.74% higher than that obtained using methods with a constant time interval and number of events, respectively. The average detection accuracy of pseudo-frame images in the YOLOE network designed in this paper is 85.11%, and the number of detection frames per second is 109. Therefore, the effectiveness of the proposed visualization method and the good performance of the designed detection network are verified.

Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks

Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.

An Interface Platform for Robotic Neuromorphic Systems

Neuromorphic computing is promising to become a future standard in low-power AI applications. The integration between new neuromorphic hardware and traditional microcontrollers is an open challenge. In this paper, we present an interface board and a communication protocol that allows communication between different devices, using a microcontroller unit (Arduino Due) in the middle. Our compact printed circuit board (PCB) links different devices as a whole system and provides a power supply for the entire system using batteries as the power supply. Concretely, we have connected a Dynamic Vision Sensor (DVS128), SpiNNaker board and a servo motor, creating a platform for a neuromorphic robotic system controlled by a Spiking Neural Network, which is demonstrated on the task of intercepting incoming objects. The data rate of the implemented interface board is 24.64 k symbols/s and the latency for generating commands is about 11ms. The complete system is run only by batteries, making it very suitable for robotic applications.

Development of DVS evaluation methods from user perspective

We report measurement methods and metrics for the evaluation of dynamic vision sensor (DVS) pixels. In particular, we developed automated test environments and test metrics which can quantify the sensitivity, latency and background noise of DVS pixels. For sensitivity measurements, response probabilities of pixels were analyzed at various conditions, such as base light intensity and region of interests of a sensor. Pixel latency was measured by varying the duty of light pulse, and noise level were also characterized at different light intensities. We expect the developed methods and metrics can help to clarify the performance of DVS pixels at the user point of view.

Low Cost and Latency Event Camera Background Activity Denoising.

Dynamic Vision Sensor (DVS) event camera output includes uninformative background activity (BA) noise events that increase dramatically under dim lighting. Existing denoising algorithms are not effective under these high noise conditions. Furthermore, it is difficult to quantitatively compare algorithm accuracy. This paper proposes a novel framework to better quantify BA denoising algorithms by measuring receiver operating characteristics with known mixtures of signal and noise DVS events. New datasets for stationary and moving camera applications of DVS in surveillance and driving are used to compare 3 new low-cost algorithms: Algorithm 1 checks distance to past events using a tiny fixed size window and removes most of the BA while preserving most of the signal for stationary camera scenarios. Algorithm 2 uses a memory proportional to the number of pixels for improved correlation checking. Compared with existing methods, it removes more noise while preserving more signal. Algorithm 3 uses a lightweight multilayer perceptron classifier driven by local event time surfaces to achieve the best accuracy over all datasets. The code and data are shared with the paper as DND21.

Distractor-aware Event-based Tracking.

Event cameras, or dynamic vision sensors, have recently achieved success from fundamental vision tasks to high-level vision researches. Due to its ability to asynchronously capture light intensity changes, event camera has an inherent advantage to capture moving objects in challenging scenarios including objects under low light, high dynamic range, or fast moving objects. Thus event camera are natural for visual object tracking. However, the current event-based trackers derived from RGB trackers simply modify the input images to event frames and still follow conventional tracking pipeline that mainly focus on object texture for target distinction. As a result, the trackers may not be robust dealing with challenging scenarios such as moving cameras and cluttered foreground. In this paper, we propose a distractor-aware event-based tracker that introduces transformer modules into Siamese network architecture (named DANet). Specifically, our model is mainly composed of a motion-aware network and a target-aware network, which simultaneously exploits both motion cues and object contours from event data, so as to discover motion objects and identify the target object by removing dynamic distractors. Our DANet can be trained in an end-to-end manner without any post-processing and can run at over 80 FPS on a single V100. We conduct comprehensive experiments on two large event tracking datasets to validate the proposed model. We demonstrate that our tracker has superior performance against the state-of-the-art trackers in terms of both accuracy and efficiency.

Ultra-High Temporal Resolution Visual Reconstruction From a Fovea-Like Spike Camera via Spiking Neuron Model.

Neuromorphic vision sensor is a new bio-inspired imaging paradigm emerged in recent years. It uses the asynchronous spike signals instead of the traditional frame-based manner to achieve ultra-high speed sampling. Unlike the dynamic vision sensor (DVS) that perceives movement by imitating the retinal periphery, the spike camera was developed recently to perceive fine textures by simulating a small retinal region called the fovea. For this new type of neuromorphic camera, how to reconstruct ultra-high speed visual images from spike data becomes an important yet challenging issue in visual scene perception, analysis, and recognition applications. In this paper, a bio-inspired visual reconstruction framework for the spike camera is proposed for the first time. Its core idea is to use the biologically inspired adaptive adjustment mechanisms, combined with the spatiotemporal spike information extracted by the proposed model, to reconstruct the full texture of natural scenes in an ultra-high temporal resolution. Specifically, the proposed model consists of a motion local excitation layer, a spike refining layer and a visual reconstruction layer motivated by the bio-realistic leaky integrate-and-fire (LIF) neurons and synapse connection with spike-timing dependent plasticity (STDP) rule. To evaluate the performance, a spike dataset was constructed for normal and high-speed scenes in real-world recorded by the spike camera. The experimental results show that the proposed approach can reconstruct the visual images with 40,000 frames per second in both normal and high-speed scenes, while achieving high dynamic range and high image quality.

TCN-CUTIE: A 1,036-TOp/s/W, 2.72-µJ/Inference, 12.2-mW All-Digital Ternary Accelerator in 22-nm FDX Technology

Tiny machine learning (TinyML) applications impose µJ/inference constraints, with a maximum power consumption of tens of megawatt. It is extremely challenging to meet these requirements at a reasonable accuracy level. This work addresses the challenge with a flexible, fully digital ternary neural network (TNN) accelerator in a reduced instruction set computer-five (RISC-V)-based System-on-Chip (SoC). Besides supporting ternary convolutional neural networks, we introduce extensions to the accelerator design that enable the processing of time-dilated temporal convolutional neural networks (TCNs). The design achieves 5.5-µJ/inference, 12.2 mW, 8,000 inferences/s at 0.5 V for a dynamic vision sensor (DVS)-based TCN and an accuracy of 94.5%, and 2.72-µJ/inference, 12.2 mW, 3,200 inferences/s at 0.5 V for a nontrivial 9-layer, 96 channels-per-layer convolutional network with CIFAR-10 accuracy of 86%. The peak energy efficiency is 1,036 TOp/s/W, outperforming the state-of-the-art silicon-proven TinyML quantized accelerators by 1.67× while achieving competitive accuracy.

Adversarial attacks on spiking convolutional neural networks for event-based vision.

Event-based dynamic vision sensors provide very sparse output in the form of spikes, which makes them suitable for low-power applications. Convolutional spiking neural networks model such event-based data and develop their full energy-saving potential when deployed on asynchronous neuromorphic hardware. Event-based vision being a nascent field, the sensitivity of spiking neural networks to potentially malicious adversarial attacks has received little attention so far. We show how white-box adversarial attack algorithms can be adapted to the discrete and sparse nature of event-based visual data, and demonstrate smaller perturbation magnitudes at higher success rates than the current state-of-the-art algorithms. For the first time, we also verify the effectiveness of these perturbations directly on neuromorphic hardware. Finally, we discuss the properties of the resulting perturbations, the effect of adversarial training as a defense strategy, and future directions.

N-Omniglot, a large-scale neuromorphic dataset for spatio-temporal sparse few-shot learning

Few-shot learning (learning with a few samples) is one of the most important cognitive abilities of the human brain. However, the current artificial intelligence systems meet difficulties in achieving this ability. Similar challenges also exist for biologically plausible spiking neural networks (SNNs). Datasets for traditional few-shot learning domains provide few amounts of temporal information. And the absence of neuromorphic datasets has hindered the development of few-shot learning for SNNs. Here, to the best of our knowledge, we provide the first neuromorphic dataset for few-shot learning using SNNs: N-Omniglot, based on the Dynamic Vision Sensor. It contains 1,623 categories of handwritten characters, with only 20 samples per class. N-Omniglot eliminates the need for a neuromorphic dataset for SNNs with high spareness and tremendous temporal coherence. Additionally, the dataset provides a powerful challenge and a suitable benchmark for developing SNNs algorithms in the few-shot learning domain due to the chronological information of strokes. We also provide the improved nearest neighbor, convolutional network, SiameseNet, and meta-learning algorithm in the spiking version for verification.