Neuromorphic Vision Sensors Research Articles

Ultrasensitive dim-light neuromorphic vision sensing via momentum-conserved reconfigurable van der Waals heterostructure

Reconfigurable phototransistors featuring bipolar photoresponses are favorable for manipulating high-performance neuromorphic vision sensory. Here, we present a momentum-conserved reconfigurable phototransistor based on the van der Waals heterojunction between methylammonium lead iodide perovskite and two-dimensional Bi2O2Se semiconductor, which exhibits a synergistic interplay of interband hot-carrier transitions and reconfigurable heterointerface band alignments, eventually achieving the ultrahigh bipolar optoelectronic performances with the photoresponsivity of 6×107 AW−1, accompanied by the specific detectivity of 5.2×1011 Jones, and the dynamic range of 110 dB. Moreover, A 3×3 heterotransistor array is fabricated to perform in-sensor analog multiply-accumulate operations even under the challenging dim illumination of 0.1 μWcm−2 that comparable to natural moonlight. The reconfigurable heterotransistor array can be further adopted to enhance the traffic-light detection under dim-light conditions. Our advancement in momentum-conserved reconfigurable heterotransistor signifies a leap forward in real-time, energy-efficient, and low-light image processing for neuromorphic vision sensors.

Two-in-one functionality in a 28 × 28 β-Ga2O3 array: bias-voltage switching between photodetection and neuromorphic vision

Due to the differences in photoresponse characteristics between photodetectors and neuromorphic vision sensors (NVS), simultaneously achieving these two powerful functionalities on a single device poses significant challenges. Here, we demonstrate a two-in-one platform based on a 28 × 28 β-Ga2O3 array that seamlessly switches between photodetector and NVS modes via bias voltage control. By exploiting the differential carrier capture dynamics of deep-level oxygen vacancies in Ga2O3, our device exhibits conventional photoconductivity at low voltages and persistent photoconductivity at high voltages. This enables high-quality optoelectronic imaging as well as excellent image sensing, memory, and neuromorphic visual preprocessing capabilities within a single integrated platform. This work paves the way for multifunctional Ga2O3 optoelectronic devices with applications in integrated sensing and computing.

Object Motion Detection Enabled by Reconfigurable Neuromorphic Vision Sensor under Ferroelectric Modulation.

Increasing the demand for object motion detection (OMD) requires shifts of reducing redundancy, heightened power efficiency, and precise programming capabilities to ensure consistency and accuracy. Drawing inspiration from object motion-sensitive ganglion cells, we propose an OMD vision sensor with a simple device structure of a WSe2 homojunction modulated by a ferroelectric copolymer. Under optical mode and intermediate ferroelectric modulation, the vision sensor can generate progressive and bidirectional photocurrents with discrete multistates under zero power consumption. This design enables reconfigurable devices to emulate long-term potentiation and depression for synaptic weights updating, which exhibit 82 states (more than 6 bits) with a uniform step of 6 pA. Such OMD devices also demonstrate nonvolatility, reversibility, symmetry, and ultrahigh linearity, achieving a fitted R2 of 0.999 and nonlinearity values of 0.01/-0.01. Thus, a vision sensor could implement motion detection by sensing only dynamic information based on the brightness difference between frames, while eliminating redundant data from static scenes. Additionally, the neural network utilizing a linear result can recognize the essential moving information with a high recognition accuracy of 96.8%. We also present the scalable potential via a uniform 3 × 3 neuromorphic vision sensor array. Our work offers a platform to achieve motion detection based on controllable and energy-efficient ferroelectric programmability.

A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-Time Systems.

Deep learning inference that needs to largely take place on the "edge" is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this article proposes a real-time, hybrid neuromorphic framework for object tracking and classification using event-based cameras that possess desirable properties such as low-power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking scheme is implemented using the apparent object velocity while tackling occlusion scenarios. The frame-based object track input is converted back to spikes for TrueNorth (TN) classification via the energy-efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TN model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the ability of our system to handle practical surveillance scenarios. As an alternative tracker paradigm, we also propose a continuous-time tracker with C++ implementation where each event is processed individually, which better exploits the low latency and asynchronous nature of neuromorphic vision sensors. Subsequently, we extensively compare the proposed methodologies to state-of-the-art event-based and frame-based methods for object tracking and classification, and demonstrate the use case of our neuromorphic approach for real-time and embedded applications without sacrificing performance. Finally, we also showcase the efficacy of the proposed neuromorphic system to a standard RGB camera setup when simultaneously evaluated over several hours of traffic recordings.

Improving Multiple Dense Prediction Performances by Exploiting Inter-Task Synergies for Neuromorphic Vision Sensors

Improving Multiple Dense Prediction Performances by Exploiting Inter-Task Synergies for Neuromorphic Vision Sensors

Unlocking Neuromorphic Vision: Advancements in IGZO-Based Optoelectronic Memristors with Visible Range Sensitivity.

Optoelectronic memristors based on amorphous oxide semiconductors (AOSs) are promising devices for the development of spiking neural network (SNN) hardware in neuromorphic vision sensors. In such devices, the conductance state can be controlled by both optical and electrical stimuli, while the typical persistent photoconductivity (PPC) of AOS materials can be used to emulate synaptic functions. However, due to the large band gap of these materials, sensitivity to visible light (red/green/blue) is difficult to accomplish, which hinders applications requiring color discrimination. In this work, we report a 4 μm2 hydrogen-doped (H-doped) indium-gallium-zinc oxide (IGZO) optoelectronic memristor that emulates all of the important rules of SNNs such as short- to long-term memory transition (STM-LTM), paired-pulse facilitation (PPF), spike-time-dependent plasticity (STDP), and learning and forgetting capabilities. By the incorporation of hydrogen gas in the sputtering deposition of IGZO, visible sensitivity was achieved for green and blue wavelengths. Additionally, extremely high light/dark ratios of 179, 93, and 12 are demonstrated for wavelengths of 365, 405, and 505 nm, respectively, due to hydrogen-induced subgap states and device miniaturization. Therefore, the proposed device shows remarkable potential for integration with the pixel circuits of IGZO-based displays with extreme resolution for a true intelligent self-processing display.

Recent advances in imaging devices: image sensors and neuromorphic vision sensors

Recent advances in imaging devices: image sensors and neuromorphic vision sensors

The role of neuromorphic and biomimetic sensors

PurposeThe purpose of this paper is to provide details of biomimetic and neuromorphic sensor research and developments and discuss their applications in robotics.Design/methodology/approachFollowing a short introduction, this first provides examples of recent biomimetic gripping and sensing skin research and developments. It then considers neuromorphic vision sensing technology and its potential robotic applications. Finally, brief conclusions are drawn.FindingsBiomimetics aims to exploit mechanisms, structures and signal processing techniques which occur in the natural world. Biomimetic sensors and control techniques can impart robots with a range of enhanced capabilities such as learning, gripping and multidimensional tactile sensing. Neuromorphic vision sensors offer several key operation benefits over conventional frame-based imaging techniques. Robotic applications are still largely at the research stage but uses are anticipated in enhanced safety systems in autonomous vehicles and in robotic gripping.Originality/valueThis illustrates how tactile and imaging sensors based on biological principles can contribute to imparting robots with enhanced capabilities.

Microsaccade-inspired event camera for robotics.

Neuromorphic vision sensors or event cameras have made the visual perception of extremely low reaction time possible, opening new avenues for high-dynamic robotics applications. These event cameras' output is dependent on both motion and texture. However, the event camera fails to capture object edges that are parallel to the camera motion. This is a problem intrinsic to the sensor and therefore challenging to solve algorithmically. Human vision deals with perceptual fading using the active mechanism of small involuntary eye movements, the most prominent ones called microsaccades. By moving the eyes constantly and slightly during fixation, microsaccades can substantially maintain texture stability and persistence. Inspired by microsaccades, we designed an event-based perception system capable of simultaneously maintaining low reaction time and stable texture. In this design, a rotating wedge prism was mounted in front of the aperture of an event camera to redirect light and trigger events. The geometrical optics of the rotating wedge prism allows for algorithmic compensation of the additional rotational motion, resulting in a stable texture appearance and high informational output independent of external motion. The hardware device and software solution are integrated into a system, which we call artificial microsaccade-enhanced event camera (AMI-EV). Benchmark comparisons validated the superior data quality of AMI-EV recordings in scenarios where both standard cameras and event cameras fail to deliver. Various real-world experiments demonstrated the potential of the system to facilitate robotics perception both for low-level and high-level vision tasks.

Neuromorphic Vision Sensor driven by Ferroelectric HfAlO

The rapid progress in bioinspired machine vision supported by intelligent computing has attracted attention of the modern electronics due to their wide applications in image recognition, artificial intelligence, and medical applications. The ferroelectric materials are considered as most reliable for the fabrication of these devices due to their nonvolatile and dense memory capacity as well as stability. Here, we report the neuromorphic vision sensor (NVS) based on a self-grown ferroelectric interfacial HfAlO layer. The unique properties of graphene (Gr) and silicon are integrated through the ferroelectric layer to fabricate the NVS, which can mimic the human eye system. The sensor efficiently demonstrates the basic neural functions including pair-pulsed facilitation, depression, inhibition, and excitation under electric as well as optical stimuli. The optical spike rate (2 Hz–10 Hz) as well as the amplitude (0.5 μW/cm2-1.5 μW/cm2) dependent plasticity, exhibited excellent facilitation and depression index of 1.62% and −1.86% with the minimum energy consumption of only 3.5 fJ per spike. The high endurance of 1 × 107 cycles confirm its reliability and data security. Our device with excellent features of optical signal detection, information processing, and data storage with its simple structure can be efficiently utilized as an image sensor for robotics and intelligent machine vision systems.

Environmentally friendly and degradable organic neuromorphic vision sensors

Environmentally friendly and degradable organic neuromorphic vision sensors

Lossless Encoding of Time-Aggregated Neuromorphic Vision Sensor Data Based on Point-Cloud Compression.

Neuromorphic Vision Sensors (NVSs) are emerging sensors that acquire visual information asynchronously when changes occur in the scene. Their advantages versus synchronous capturing (frame-based video) include a low power consumption, a high dynamic range, an extremely high temporal resolution, and lower data rates. Although the acquisition strategy already results in much lower data rates than conventional video, NVS data can be further compressed. For this purpose, we recently proposed Time Aggregation-based Lossless Video Encoding for Neuromorphic Vision Sensor Data (TALVEN), consisting in the time aggregation of NVS events in the form of pixel-based event histograms, arrangement of the data in a specific format, and lossless compression inspired by video encoding. In this paper, we still leverage time aggregation but, rather than performing encoding inspired by frame-based video coding, we encode an appropriate representation of the time-aggregated data via point-cloud compression (similar to another one of our previous works, where time aggregation was not used). The proposed strategy, Time-Aggregated Lossless Encoding of Events based on Point-Cloud Compression (TALEN-PCC), outperforms the originally proposed TALVEN encoding strategy for the content in the considered dataset. The gain in terms of the compression ratio is the highest for low-event rate and low-complexity scenes, whereas the improvement is minimal for high-complexity and high-event rate scenes. According to experiments on outdoor and indoor spike event data, TALEN-PCC achieves higher compression gains for time aggregation intervals of more than 5 ms. However, the compression gains are lower when compared to state-of-the-art approaches for time aggregation intervals of less than 5 ms.

A Multimode Neuromorphic Vision Sensor With Improved Brightness Measurement Performance by Pulse Coding Method

A Multimode Neuromorphic Vision Sensor With Improved Brightness Measurement Performance by Pulse Coding Method

Monolithic 2D Perovskites Enabled Artificial Photonic Synapses for Neuromorphic Vision Sensors.

Neuromorphic visual sensors (NVS) based on photonic synapses hold a significant promise to emulate the human visual system. However, current photonic synapses rely on exquisite engineering of the complex heterogeneous interface to realize learning and memory functions, resulting in high fabrication cost, reduced reliability, high energy consumption and uncompact architecture, severely limiting the up-scaled manufacture, and on-chip integration. Here a photo-memory fundamental based on ion-exciton coupling is innovated to simplify synaptic structure and minimize energy consumption. Due to the intrinsic organic/inorganic interface within the crystal, the photodetector based on monolithic 2D perovskite exhibits a persistent photocurrent lasting about 90s, enabling versatile synaptic functions. The electrical power consumption per synaptic event is estimated to be≈1.45×10-16J, one order of magnitude lower than that in a natural biological system. Proof-of-concept image preprocessing using the neuromorphic vision sensors enabled by photonic synapse demonstrates 4 times enhancement of classification accuracy. Furthermore, getting rid of the artificial neural network, an expectation-based thresholding model is put forward to mimic the human visual system for facial recognition. This conceptual device unveils a new mechanism to simplify synaptic structure, promising the transformation of the NVS and fostering the emergence of next generation neural networks.

In situ cryptography in a neuromorphic vision sensor based on light-driven memristors

Vision sensors are becoming increasingly ubiquitous, and they continuously collect, store, communicate, and process vast amount of sensitive data that are vulnerable to being stolen and misused. Existing cryptosystems based on complex cipher algorithms generally require extensive computational resources, making them difficult to use in vision sensors that have limited processing capabilities. Here, we propose and experimentally demonstrate a novel in situ image cryptography scheme based on a neuromorphic vision sensor comprising all-optically controlled (AOC) memristors. Due to the unique light wavelength and irradiation history-dependent bidirectional persistent photoconductivity of AOC memristors, a visual image can be stored, encrypted, decrypted, denoised, and destroyed within a vision sensor. A decrypted image can be encoded in situ and then accurately recognized through a memristive neural network. Encrypted and destroyed images are capable of withstanding hacking attacks even with trained neural networks. Our cryptography scheme enables complete cryptographic operations entirely on a sensor and, therefore, effectively safeguards visual information. This work provides a simple yet efficient solution to the security challenges faced by vision sensors.

Memory effect and coexistence of negative and positive photoconductivity in black phosphorus field effect transistor for neuromorphic vision sensors.

Black phosphorus (BP) field-effect transistors with ultrathin channels exhibit unipolar p-type electrical conduction over a wide range of temperatures and pressures. Herein, we study a device that exhibits mobility up to 100 cm2 V-1 s-1 and a memory window up to 1.3 μA. Exposure to a supercontinuum white light source reveals that negative photoconductivity (NPC) and positive photoconductivity (PPC) coexist in the same device. Such behavior is attributed to the chemisorbed O2 molecules, with a minor role of physisorbed H2O molecules. The coexistence of NPC and PPC can be exploited in neuromorphic vision sensors, requiring the human eye retina to process the optical signals through alerting and protection (NPC), adaptation (PPC), followed by imaging and processing. Our results open new avenues for the use of BP and other two-dimentional (2D) semiconducting materials in transistors, memories, and neuromorphic vision sensors for advanced applications in robotics, self-driving cars, etc.

Optoelectronic stimuli-driven switchable memristors with multilevel resistance states for neuromorphic vision sensors

The developed optoelectronic synaptic devices with resistive switching enable efficient, fault-resistant image recognition. The PMMA/ZnO/2IF memristor array advances neuromorphic vision systems, offering high accuracy and low power consumption.

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

Introduction: Advances in machine vision and mobile electronics will be accelerated by the creation of sophisticated optoelectronic vision sensors that allow for sophisticated picture recognition of visual information and data pre-processing. Several new types of vision sensors have been devised in the last decade to solve these drawbacks, one of which is neuromorphic vision sensors, which have exciting qualities such as high temporal resolution, broad dynamic range, and low energy consumption. Neuromorphic sensors are inspired by the working principles of biological sensory neurons and would be useful in telemedicine, health surveillance, security monitoring, automatic driving, intelligent robots, and other applications of the Internet of Things.Methods: This paper provides a comprehensive review of various state-of-the-art AI vision sensors and frameworks.Results: The fundamental signal processing techniques deployed and the associated challenges were discussed.Discussion: Finally, the role of vision sensors in computer vision is also discussed.

Widefield Diamond Quantum Sensing with Neuromorphic Vision Sensors.

Despite increasing interest in developing ultrasensitive widefield diamond magnetometry for various applications, achieving high temporal resolution and sensitivity simultaneously remains a key challenge. This is largely due to the transfer and processing of massive amounts of data from the frame-based sensor to capture the widefield fluorescence intensity of spin defects in diamonds. In this study, a neuromorphic vision sensor to encode the changes of fluorescence intensity into spikes in the optically detected magnetic resonance (ODMR) measurements is adopted, closely resembling the operation of the human vision system, which leads to highly compressed data volume and reduced latency. It also results in a vast dynamic range, high temporal resolution, and exceptional signal-to-background ratio. After a thorough theoretical evaluation, the experiment with an off-the-shelf event camera demonstrated a 13× improvement in temporal resolution with comparable precision of detecting ODMR resonance frequencies compared with the state-of-the-art highly specialized frame-based approach. It is successfully deploy this technology in monitoring dynamically modulated laser heating of gold nanoparticles coated on a diamond surface, a recognizably difficult task using existing approaches. The current development provides new insights for high-precision and low-latency widefield quantum sensing, with possibilities for integration with emerging memory devices to realize more intelligent quantum sensors.

Retina-inspired organic neuromorphic vision sensor with polarity modulation for decoding light information

The neuromorphic vision sensor (NeuVS), which is based on organic field-effect transistors (OFETs), uses polar functional groups (PFGs) in polymer dielectrics as interfacial units to control charge carriers. However, the mechanism of modulating charge transport on basis of PFGs in devices is unclear. Here, the carboxyl group is introduced into polymer dielectrics in this study, and it can induce the charge transfer process at the semiconductor/dielectric interfaces for effective carrier transport, giving rise to the best device mobility up to 20 cm2 V−1 s−1 at a low operating voltage of −1 V. Furthermore, the polarity modulation effect could further increase the optical figures of merit in NeuVS devices by at least an order of magnitude more than the devices using carboxyl group-free polymer dielectrics. Additionally, devices containing carboxyl groups improved image sensing for light information decoding with 52 grayscale signals and memory capabilities at an incredibly low power consumption of 1.25 fJ/spike. Our findings provide insight into the production of high-performance polymer dielectrics for NeuVS devices.