Neuromorphic Sensors Research Articles

Active fixation as an efficient coding strategy for neuromorphic vision

Contrary to a photographer, who puts a great effort in keeping the lens still, eyes insistently move even during fixation. This benefits signal decorrelation, which underlies an efficient encoding of visual information. Yet, camera motion is not sufficient alone; it must be coupled with a sensor specifically selective to temporal changes. Indeed, motion induced on standard imagers only results in burring effects. Neuromorphic sensors represent a valuable solution. Here we characterize the response of an event-based camera equipped with fixational eye movements (FEMs) on both synthetic and natural images. Our analyses prove that the system starts an early stage of redundancy suppression, as a precursor of subsequent whitening processes on the amplitude spectrum. This does not come at the price of corrupting structural information contained in local spatial phase across oriented axes. Isotropy of FEMs ensures proper representations of image features without introducing biases towards specific contrast orientations.

Tetrachromatic vision-inspired neuromorphic sensors with ultraweak ultraviolet detection

Sensing and recognizing invisible ultraviolet (UV) light is vital for exploiting advanced artificial visual perception system. However, due to the uncertainty of the natural environment, the UV signal is very hard to be detected and perceived. Here, inspired by the tetrachromatic visual system, we report a controllable UV-ultrasensitive neuromorphic vision sensor (NeuVS) that uses organic phototransistors (OPTs) as the working unit to integrate sensing, memory and processing functions. Benefiting from asymmetric molecular structure and unique UV absorption of the active layer, the as fabricated UV-ultrasensitive NeuVS can detect 370 nm UV-light with the illumination intensity as low as 31 nW cm−2, exhibiting one of the best optical figures of merit in UV-sensitive neuromorphic vision sensors. Furthermore, the NeuVS array exbibits good image sensing and memorization capability due to its ultrasensitive optical detection and large density of charge trapping states. In addition, the wavelength-selective response and multi-level optical memory properties are utilized to construct an artificial neural network for extract and identify the invisible UV information. The NeuVS array can perform static and dynamic image recognition from the original color image by filtering red, green and blue noise, and significantly improve the recognition accuracy from 46 to 90%.

Carbon Nanodots Memristor: An Emerging Candidate toward Artificial Biosynapse and Human Sensory Perception System.

In the era of big data and artificial intelligence (AI), advanced data storage and processing technologies are in urgent demand. The innovative neuromorphic algorithm and hardware based on memristor devices hold a promise to break the von Neumann bottleneck. In recent years, carbon nanodots (CDs) have emerged as a new class of nano-carbon materials, which have attracted widespread attention in the applications of chemical sensors, bioimaging, and memristors. The focus of this review is to summarize the main advances of CDs-based memristors, and their state-of-the-art applications in artificial synapses, neuromorphic computing, and human sensory perception systems. The first step is to systematically introduce the synthetic methods of CDs and their derivatives, providing instructive guidance to prepare high-quality CDs with desired properties. Then, the structure-property relationship and resistive switching mechanism of CDs-based memristors are discussed in depth. The current challenges and prospects of memristor-based artificial synapses and neuromorphic computing are also presented. Moreover, this review outlines some promising application scenarios of CDs-based memristors, including neuromorphic sensors and vision, low-energy quantum computation, and human-machine collaboration.

Neuromorphic Wireless Cognition: Event-Driven Semantic Communications for Remote Inference

Neuromorphic computing is an emerging computing paradigm that moves away from batched processing towards the online, event-driven, processing of streaming data. Neuromorphic chips, when coupled with spike-based sensors, can inherently adapt to the “semantics” of the data distribution by consuming energy only when relevant events are recorded in the timing of spikes and by proving a low-latency response to changing conditions in the environment. This paper proposes an end-to-end design for a neuromorphic wireless Internet-of-Things system that integrates spike-based sensing, processing, and communication. In the proposed NeuroComm system, each sensing device is equipped with a neuromorphic sensor, a spiking neural network (SNN), and an impulse radio (IR) transmitter with multiple antennas. Transmission takes place over a shared fading channel to a receiver equipped with a multi-antenna impulse radio receiver and with an SNN. In order to enable adaptation of the receiver to the fading channel conditions, we introduce a hypernetwork to control the weights of the decoding SNN using pilots. Pilots, encoding SNNs, decoding SNN, and hypernetwork are jointly trained across multiple channel realizations. The proposed system is shown to significantly improve over conventional frame-based digital solutions, as well as over alternative non-adaptive training methods, in terms of time-to-accuracy and energy consumption metrics.

Neuromorphic Sensor Event-Rate Monitoring for Satellite Characterization

A neuromorphic event-based camera was fitted to Electro Optic Systems’s 1.8 m telescope at Mount Stromlo in Canberra and used to observe 14 satellites in a variety of orbit regimes. Unique patterns in the fluctuations of each satellite’s event rate were analyzed to assess the utility of event-based sensors for satellite characterization. Event-rate curves were shown to be useful for determining the rotation rate of nonstabilized satellites, constraining the range of possible satellite orientations, and for differentiating between materials responsible for bright specular reflections in some cases.

QISampling: An Effective Sampling Strategy for Event-Based Sign Language Recognition

Event cameras are innovative neuromorphic sensors that detect changes in brightness and generate a stream of events, thus providing high temporal resolution and dynamic range advantages. With its ability to perceive motion information, event data is well-suited for sign language recognition (SLR). Existing methods of event-based SLR rely on a uniform sampling strategy, which may result in redundant and indiscriminate information being captured when segments are randomly sampled. In this letter, we propose an effective sampling strategy called Quantity-inspired Sampling (QISampling) that takes advantage of the quantity feature of event distribution to sample key segments containing more discriminative and remarkable motion features from a sign. These segments are then converted into frame-like representations, which serve as input to subsequent Deep Neural Networks (DNNs). In addition, we introduce a synthetic event-based sign language dataset N-WLASL to the community. We apply our strategy to DNNs and conduct experiments on this synthetic dataset and another real-world dataset. The results demonstrate that our QISampling strategy could accelerate training and make event data provide superior performance for the SLR task.

Adaptive Memory and In Materia Reinforcement Learning Enabled by Flexoelectric-like Response from Ultrathin HfO2.

Reinforcement learning (RL) is a mathematical framework of neural learning by trial and error that revolutionized the field of artificial intelligence. However, until now, RL has been implemented in algorithms with the compatibly of traditional complementary metal-oxide-semiconductor-based von Neumann digital platforms, which thus limits performance in terms of latency, fault tolerance, and robustness. Here, we demonstrate that nanocolumnar (∼12 nm) HfO2 structures can be used as building blocks to conduct the RL within the material by combining its stress-adjustable charge transport and memory functions. Specifically, HfO2 nanostructures grown by the sputtering method exhibit self-assembled vertical nanocolumnar structures that generate voltage depending on the impact of stress under self-biased conditions. The observed results are attributed to the flexoelectric-like response of HfO2. Further, multilevel current (>10-3 A) modulation with touch and controlled suspension (∼10-12 A) with a short electric pulse (100 ms) were demonstrated, yielding a proof-of-concept memory with an on/off ratio greater than 109. Utilizing multipattern dynamic memory and tactile sensing, RL was implemented to successfully solve a maze game using an array of 6 × 4. This work could pave the way to do RL within materials for a variety of applications such as memory storage, neuromorphic sensors, smart robots, and human-machine interaction systems.

EBBINNOT: A Hardware-Efficient Hybrid Event-Frame Tracker for Stationary Dynamic Vision Sensors

As an alternative sensing paradigm, dynamic vision sensors (DVSs) have been recently explored to tackle scenarios where conventional sensors result in high data rate and processing time. This article presents a hybrid event-frame approach for detecting and tracking objects recorded by a stationary neuromorphic sensor, thereby exploiting the sparse DVS output in a low-power setting for traffic monitoring. Specifically, we propose a hardware-efficient processing pipeline that optimizes memory and computational needs that enable long-term battery-powered usage for Internet of Things applications. To exploit the background removal property of a static DVS, we propose an event-based binary image creation that signals presence or absence of events in a frame duration. This reduces memory requirement and enables the usage of simple algorithms like median filtering and connected component labeling for denoise and region proposal (RP), respectively. To overcome the fragmentation issue, a YOLO-inspired neural network-based detector and classifier to merge fragmented RPs has been proposed. Finally, a new overlap-based tracker was implemented, exploiting overlap between detections and tracks is proposed with heuristics to overcome occlusion. The proposed pipeline is evaluated with more than 5 h of traffic recording spanning three different locations on two different neuromorphic sensors (DVS and CeleX) and demonstrates similar performance. Compared to existing event-based feature trackers, our method provides similar accuracy while needing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\approx {6}\times $ </tex-math></inline-formula> less computes. To the best of our knowledge, this is the first time a stationary DVS-based traffic monitoring solution is extensively compared to simultaneously recorded RGB frame-based methods while showing tremendous promise by outperforming state-of-the-art deep learning solutions. The traffic data set is publicly made available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://nusneuromorphic.github.io/dataset/index.html</uri> .

Event-Based Angular Speed Measurement and Movement Monitoring.

Computer vision techniques can monitor the rotational speed of rotating equipment or machines to understand their working conditions and prevent failures. Such techniques are highly precise, contactless, and potentially suitable for applications without massive setup changes. However, traditional vision sensors collect a significant amount of data to process and measure the rotation of high-speed systems, and they are susceptible to motion blur. This work proposes a new method for measuring rotational speed processing event-based data applied to high-speed systems using a neuromorphic sensor. This sensor produces event-based data and is designed to work with high temporal resolution and high dynamic range. The main advantages of the Event-based Angular Speed Measurement (EB-ASM) method are the high dynamic range, the absence of motion blurring, and the possibility of measuring multiple rotations simultaneously with a single device. The proposed method uses the time difference between spikes in a Kernel or Window selected in the sensor frame range. It is evaluated in two experimental scenarios by measuring a fan rotational speed and a Router Computer Numerical Control (CNC) spindle. The results compare measurements with a calibrated digital photo-tachometer. Based on the performed tests, the EB-ASM can measure the rotational speed with a mean absolute error of less than 0.2% for both scenarios.

Real-Time Optical Flow for Vehicular Perception With Low- and High-Resolution Event Cameras

Event cameras capture changes of illumination in the observed scene rather than accumulating light to create images. Thus, they allow for applications under high-speed motion and complex lighting conditions, where traditional frame-based sensors show their limits with blur and over- or under-exposed pixels. Thanks to these unique properties, they represent nowadays an highly attractive sensor for ITS-related applications. Event-based optical flow (EBOF) has been studied following the rise in popularity of these neuromorphic cameras. The recent arrival of high-definition neuromorphic sensors, however, challenges the existing approaches, because of the increased resolution of the events pixel array and a much higher throughput. As an answer to these points, we propose an optimized framework for computing optical flow in real-time with both low- and high-resolution event cameras. We formulate a novel dense representation for the sparse events flow, in the form of the “inverse exponential distance surface”. It serves as an interim frame, designed for the use of proven, state-of-the-art frame-based optical flow computation methods. We evaluate our approach on both low- and high-resolution driving sequences, and show that it often achieves better results than the current state of the art, while also reaching higher frame rates, 250Hz at 346 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times260$ </tex-math></inline-formula> pixels and 77Hz at 1280 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times720$ </tex-math></inline-formula> pixels.

EventHD: Robust and efficient hyperdimensional learning with neuromorphic sensor.

Brain-inspired computing models have shown great potential to outperform today's deep learning solutions in terms of robustness and energy efficiency. Particularly, Hyper-Dimensional Computing (HDC) has shown promising results in enabling efficient and robust cognitive learning. In this study, we exploit HDC as an alternative computational model that mimics important brain functionalities toward high-efficiency and noise-tolerant neuromorphic computing. We present EventHD, an end-to-end learning framework based on HDC for robust, efficient learning from neuromorphic sensors. We first introduce a spatial and temporal encoding scheme to map event-based neuromorphic data into high-dimensional space. Then, we leverage HDC mathematics to support learning and cognitive tasks over encoded data, such as information association and memorization. EventHD also provides a notion of confidence for each prediction, thus enabling self-learning from unlabeled data. We evaluate EventHD efficiency over data collected from Dynamic Vision Sensor (DVS) sensors. Our results indicate that EventHD can provide online learning and cognitive support while operating over raw DVS data without using the costly preprocessing step. In terms of efficiency, EventHD provides 14.2× faster and 19.8× higher energy efficiency than state-of-the-art learning algorithms while improving the computational robustness by 5.9×.

A Bioinspired Artificial Gustatory Neuron for a Neuromorphic Based Electronic Tongue

A novel biomimicked neuromorphic sensor for an energy efficient and highly scalable electronic tongue (E-tongue) is demonstrated with a metal-oxide-semiconductor field-effect transistor (MOSFET). By mimicking a biological gustatory neuron, the proposed E-tongue can simultaneously detect ion concentrations of chemicals on an extended gate and encode spike signals on the MOSFET, which acts as an input neuron in a spiking neural network (SNN). Such in-sensor neuromorphic functioning can reduce the energy and area consumption of the conventional E-tongue hardware. pH-sensitive and sodium-sensitive artificial gustatory neurons are implemented by using two different sensing materials: Al2O3 for pH sensing and sodium ionophore X for sodium ion sensing. In addition, a sensitivity control function inspired by the biological sensory neuron is demonstrated. After the unit device characterization of the artificial gustatory neuron, a fully hardware-based E-tongue that can classify two distinct liquids is demonstrated to show a practical application of the artificial gustatory neurons.

From Theoretical Network to Bedside: Translational Application of Brain-Inspired Computing in Clinical Medicine

Advances in the brain-inspired computing space are growing at a rapid rate, and many of these emerging strategies are in the field of neuromorphic control, robotics, and sensor development, just to name a few. These innovations are disruptive in their own right and have numerous, multi-dimensional medical applications within precision medicine, telematics, device development, and informed clinical decision making. For this discussion, I will define brain-inspired computing in the scope of simulating the architecture of the brain and discuss the realization of integrating hardware and other technologies with the applications of medicine, along with the considerations for the regulatory pathway for approval and evaluating the risk/consequences of failure modes. This perspective is a call for continued discussion of the development of a pathway for translating these technologies into medical treatment and diagnostic strategies. The aim is to align with global regulatory bodies and ensure that regulation does not limit the capacity of these emerging innovations while ensuring patient safety and clinical efficacy. It is my perspective that it is and will continue to be critical that these technologies are correctly perceived and understood in the lens of multiple disciplines in order to reach their full potential for medical applications.

Event-driven proto-object based saliency in 3D space to attract a robot’s attention

To interact with its environment, a robot working in 3D space needs to organise its visual input in terms of objects or their perceptual precursors, proto-objects. Among other visual cues, depth is a submodality used to direct attention to visual features and objects. Current depth-based proto-object attention models have been implemented for standard RGB-D cameras that produce synchronous frames. In contrast, event cameras are neuromorphic sensors that loosely mimic the function of the human retina by asynchronously encoding per-pixel brightness changes at very high temporal resolution, thereby providing advantages like high dynamic range, efficiency (thanks to their high degree of signal compression), and low latency. We propose a bio-inspired bottom-up attention model that exploits event-driven sensing to generate depth-based saliency maps that allow a robot to interact with complex visual input. We use event-cameras mounted in the eyes of the iCub humanoid robot to directly extract edge, disparity and motion information. Real-world experiments demonstrate that our system robustly selects salient objects near the robot in the presence of clutter and dynamic scene changes, for the benefit of downstream applications like object segmentation, tracking and robot interaction with external objects.

Flexible VO2 Films for In‐Sensor Computing with Ultraviolet Light

AbstractWith their unique advantages in portability, shape adaptability, and human friendly surfaces, flexible electronics pave the way for the implementation of wearable electronic textiles and human–machine interfaces. Although organic materials are promising for flexible devices because of the low‐cost manufacturing and inherent flexibility, they meet challenges in harsh environments such as ultraviolet (UV) irradiation, which limits their applicability in UV sensors. Here, a flexible UV neuromorphic sensor is presented using inorganic vanadium dioxide (VO2) films grown on mica substrates. The flexible device shows UV photoinduced nonvolatile phase transition, and can be reversibly modulated using electrolyte gating. The optical responses remain almost unchanged after 10 000 bending cycles or at small bending radius, exhibiting high tolerance to the bending deformation. Besides, the variations in image recognition accuracy under different bending conditions keep within 1.6%, indicating that the device can be adapted to various deformation conditions. By constructing near‐/in‐sensor computing architectures using the flexible VO2 neuromorphic sensors with photoinduced nonvolatile phase transition, both static image processing and motion detection are realized without redundant and massive information transfer. This result lays the foundation for the development of flexible UV neuromorphic sensors.

Photo-induced non-volatile VO2 phase transition for neuromorphic ultraviolet sensors

In the quest for emerging in-sensor computing, materials that respond to optical stimuli in conjunction with non-volatile phase transition are highly desired for realizing bioinspired neuromorphic vision components. Here, we report a non-volatile multi-level control of VO2 films by oxygen stoichiometry engineering under ultraviolet irradiation. Based on the reversible regulation of VO2 films using ultraviolet irradiation and electrolyte gating, we demonstrate a proof-of-principle neuromorphic ultraviolet sensor with integrated sensing, memory, and processing functions at room temperature, and also prove its silicon compatible potential through the wafer-scale integration of a neuromorphic sensor array. The device displays linear weight update with optical writing because its metallic phase proportion increases almost linearly with the light dosage. Moreover, the artificial neural network consisting of this neuromorphic sensor can extract ultraviolet information from the surrounding environment, and significantly improve the recognition accuracy from 24% to 93%. This work provides a path to design neuromorphic sensors and will facilitate the potential applications in artificial vision systems.

Perspective of smart self-powered neuromorphic sensor and their challenges towards artificial intelligence for next-generation technology

Emerging flexible artificial sensory systems that can emulate biological sensory perception are highly desirable due to intense interest in artificial intelligence (AI) worldwide. As AI research advances, there is a growing interest in brain-inspired computing via innovative neuromorphic devices. Sensory memory, created during the early stages of perception and interaction with the environment, is regarded as the fundamental source of intelligence. Transferring these biological principles to electronic implementation aspires to achieve perceptual intelligence, significantly advancing a wide range of applications, including prosthetics and robotics. We highlight current breakthroughs in self-powered neuromorphic devices and potential challenges in the development of the perception system. It can be foreseen that artificial sensory systems will revolutionize technology and lead to a significant human–computer interaction model.

Self-Powered Sensors: New Opportunities and Challenges from Two-Dimensional Nanomaterials.

Nanomaterials have gained considerable attention over the last decade, finding applications in emerging fields such as wearable sensors, biomedical care, and implantable electronics. However, these applications require miniaturization operating with extremely low power levels to conveniently sense various signals anytime, anywhere, and show the information in various ways. From this perspective, a crucial field is technologies that can harvest energy from the environment as sustainable, self-sufficient, self-powered sensors. Here we revisit recent advances in various self-powered sensors: optical, chemical, biological, medical, and gas. A timely overview is provided of unconventional nanomaterial sensors operated by self-sufficient energy, focusing on the energy source classification and comparisons of studies including self-powered photovoltaic, piezoelectric, triboelectric, and thermoelectric technology. Integration of these self-operating systems and new applications for neuromorphic sensors are also reviewed. Furthermore, this review discusses opportunities and challenges from self-powered nanomaterial sensors with respect to their energy harvesting principles and sensing applications.

Event Camera Simulator Improvements via Characterized Parameters.

It has been more than two decades since the first neuromorphic Dynamic Vision Sensor (DVS) sensor was invented, and many subsequent prototypes have been built with a wide spectrum of applications in mind. Competing against state-of-the-art neural networks in terms of accuracy is difficult, although there are clear opportunities to outperform conventional approaches in terms of power consumption and processing speed. As neuromorphic sensors generate sparse data at the focal plane itself, they are inherently energy-efficient, data-driven, and fast. In this work, we present an extended DVS pixel simulator for neuromorphic benchmarks which simplifies the latency and the noise models. In addition, to more closely model the behaviour of a real pixel, the readout circuitry is modelled, as this can strongly affect the time precision of events in complex scenes. Using a dynamic variant of the MNIST dataset as a benchmarking task, we use this simulator to explore how the latency of the sensor allows it to outperform conventional sensors in terms of sensing speed.

Adaptive SNN for Anthropomorphic Finger Control.

Anthropomorphic hands that mimic the smoothness of human hand motions should be controlled by artificial units of high biological plausibility. Adaptability is among the characteristics of such control units, which provides the anthropomorphic hand with the ability to learn motions. This paper presents a simple structure of an adaptive spiking neural network implemented in analogue hardware that can be trained using Hebbian learning mechanisms to rotate the metacarpophalangeal joint of a robotic finger towards targeted angle intervals. Being bioinspired, the spiking neural network drives actuators made of shape memory alloy and receives feedback from neuromorphic sensors that convert the joint rotation angle and compression force into the spiking frequency. The adaptive SNN activates independent neural paths that correspond to angle intervals and learns in which of these intervals the rotation the finger rotation is stopped by an external force. Learning occurs when angle-specific neural paths are stimulated concurrently with the supraliminar stimulus that activates all the neurons that inhibit the SNN output stopping the finger. The results showed that after learning, the finger stopped in the angle interval in which the angle-specific neural path was active, without the activation of the supraliminar stimulus. The proposed concept can be used to implement control units for anthropomorphic robots that are able to learn motions unsupervised, based on principles of high biological plausibility.