Bio-inspired Vision Sensors Research Articles

Artificial Amacrine Retinal Circuits.

Event-based imaging represents a new paradigm in visual information processing that addresses the speed and energy efficiency shortcomings inherently present in the current complementary metal oxide semiconductor-based machine vision. Realizing such imaging systems has previously been sought using very large-scale integration technologies that have complex circuitries consisting of many photodiodes, differential amplifiers, capacitors, and resistors. Here, we demonstrate that event-driven sensing can be achieved using a simple one-resistor, one-capacitor (1R1C) circuit, where the capacitor is modified with colloidal quantum dots (CQDs) to have a photoresponse. This sensory circuit emulates the motion-tracking function of the biological retina, in which the amacrine cells in the bipolar-to-ganglion synaptic pathway produce a transient spiking signal only in response to changes in light intensity but remain inactive under constant illumination. When extended to a 2D imaging array, the individual sensors work independently and output signals only when a change in the light intensity is detected; hence, the concept of the frame in image processing is thereby removed. In this work, we present the fabrication and characterization of a CQD photocapacitor-based 1R1C circuit that has a spectral response at 1550 nm in the short-wave infrared (SWIR). We report on the key performance parameters including peak responsivity, noise, and optical noise equivalent power and discuss the operating mechanism that is responsible for spiking responses in these artificial retinal circuits. The present work sets the foundation for expanding the bioinspired vision sensor capability toward midwave infrared (MWIR) and long-wave infrared (LWIR) spectral regions that are invisible to human eyes and mainstream semiconductor technologies.

Recent Advances in Bio-Inspired Vision Sensor: A Review

Event-based cameras, also known as biologically inspired visual sensors, are capable of capturing real-time scene changes efficiently. Unlike traditional frame-based cameras, event cameras solely report triggered pixel-level brightness changes which are referred to as events. Event-based cameras show many advantages such as high temporal resolution, low latency, and high dynamic range, making them very attractive in robotics and computer vision, especially in challenging scenarios that are too demanding for traditional cameras. In this paper, we provide a comprehensive overview of the emerging field of event-based vision, focusing on the operation principle, sampling mechanisms, and algorithms that take advantage of their superior features. We also delve into the various tasks for which event cameras are utilized, such as object tracking, optical flow estimation, 3D reconstruction, SLAM, image reconstruction, and recognition. Additionally, we highlight the challenges and future opportunities for event cameras, seeking a more efficient way for machines to perceive and interact with the world.

Finding Visual Saliency in Continuous Spike Stream

As a bio-inspired vision sensor, the spike camera emulates the operational principles of the fovea, a compact retinal region, by employing spike discharges to encode the accumulation of per-pixel luminance intensity. Leveraging its high temporal resolution and bio-inspired neuromorphic design, the spike camera holds significant promise for advancing computer vision applications. Saliency detection mimic the behavior of human beings and capture the most salient region from the scenes. In this paper, we investigate the visual saliency in the continuous spike stream for the first time. To effectively process the binary spike stream, we propose a Recurrent Spiking Transformer (RST) framework, which is based on a full spiking neural network. Our framework enables the extraction of spatio-temporal features from the continuous spatio-temporal spike stream while maintaining low power consumption. To facilitate the training and validation of our proposed model, we build a comprehensive real-world spike-based visual saliency dataset, enriched with numerous light conditions. Extensive experiments demonstrate the superior performance of our Recurrent Spiking Transformer framework in comparison to other spike neural network-based methods. Our framework exhibits a substantial margin of improvement in capturing and highlighting visual saliency in the spike stream, which not only provides a new perspective for spike-based saliency segmentation but also shows a new paradigm for full SNN-based transformer models. The code and dataset are available at https://github.com/BIT-Vision/SVS.

Action Recognition and Benchmark Using Event Cameras.

Recent years have witnessed remarkable achievements in video-based action recognition. Apart from traditional frame-based cameras, event cameras are bio-inspired vision sensors that only record pixel-wise brightness changes rather than the brightness value. However, little effort has been made in event-based action recognition, and large-scale public datasets are also nearly unavailable. In this paper,we present an event-based action recognition framework called EV-ACT. The Learnable Multi-Fused Representation (LMFR) is first proposed to integrate multiple event information in a learnable manner. The LMFR with dual temporal granularity is fed into the event-based slow-fast network for the fusion of appearance and motion features. A spatial-temporal attention mechanism is introduced to further enhance the learning capability of action recognition. To prompt research in this direction, we have collected the largest event-based action recognition benchmark named THUE-ACT-50 and the accompanying THUE-ACT-50-CHL dataset under challenging environments, including a total of over 12,830 recordings from 50 action categories, which is over 4 times the size of the previous largest dataset. Experimental results show that our proposed framework could achieve improvements of over 14.5%, 7.6%, 11.2%, and 7.4% compared to previous works on four benchmarks. We have also deployed our proposed EV-ACT framework on a mobile platform to validate its practicality and efficiency.

A temporal densely connected recurrent network for event-based human pose estimation

Event camera is an emerging bio-inspired vision sensors that report per-pixel brightness changes asynchronously. It holds noticeable advantage of high dynamic range, high speed response, and low power budget that enable it to best capture local motions in uncontrolled environments. This motivates us to unlock the potential of event cameras for human pose estimation, as the human pose estimation with event cameras is rarely explored. Due to the novel paradigm shift from conventional frame-based cameras, however, event signals in a time interval contain very limited information, as event cameras can only capture the moving body parts and ignores those static body parts, resulting in some parts to be incomplete or even disappeared in the time interval. This paper proposes a novel densely connected recurrent architecture to address the problem of incomplete information. By this recurrent architecture, we can explicitly model not only the sequential but also non-sequential geometric consistency across time steps to accumulate information from previous frames to recover the entire human bodies, achieving a stable and accurate human pose estimation from event data. Moreover, to better evaluate our model, we collect a large-scale multimodal event-based dataset that comes with human pose annotations, which is by far the most challenging one to the best of our knowledge. The experimental results on two public datasets and our own dataset demonstrate the effectiveness and strength of our approach. Code is available online for facilitating the future research.

InP Quantum Dots Tailored Oxide Thin Film Phototransistor for Bioinspired Visual Adaptation

AbstractThe exploration of bionic neuromorphic chips, capable of processing sensory data in a human‐like manner, is both a trend and a challenge. There is a strong demand for phototransistors that offer broadband in‐sensor adaptability. This study introduces a bioinspired vision sensor based on InP quantum dots (QDs)/InSnZnO hybrid phototransistors. This novel design combines the excellent electrical transportation features of oxide semiconductors with the superior optoelectronic response of InP QDs. The resulting hybrid devices exhibit exceptional gate controllability and a robust visible‐light response. These characteristics enable the emulation of multiple functions of the human visual system and the accommodation of varying light intensity environments. Furthermore, the phototransistor array successfully replicates the scotopic and photopic adaptation recognition behaviors of the human retina. Notably, the device demonstrates faultless competency in image processing, achieving an impressive 93% accuracy for digit recognition. These findings contribute to the advancement of bionic neuromorphic chips and offer promising opportunities for future developments in the bioinspired visual system.

Recent advances in bioinspired vision sensor arrays based on advanced optoelectronic materials

Animals can learn about the outside world in many ways, and the visual organ is a key organ for acquiring information about the outside world. With the continuous development of intelligent technology, artificial vision techniques are becoming easier and more automated; however, the rigidity, process complexity, and complicated optical components of traditional commercial photodetectors have hindered their development in bionic vision. In recent years, a new generation of optoelectronic materials has attracted extensive research due to their simple preparation process, continuously tunable bandgap, and excellent optoelectronic properties. Two-dimensional optoelectronic materials and perovskites have become the most promising and effective optoelectronic materials for next-generation optoelectronic devices. Based on the excellent properties of next-generation optoelectronic materials, they have also triggered intensive exploration by researchers in the field of visual bionics. This paper highlights a review of the latest research progress of next-generation optoelectronic materials, including their preparation methods, working mechanisms, structural designs, and advances in the field of imaging. The applications of new generation optoelectronic materials in visual bionics by simulating biological visual structures are also described. Finally, the prospects and challenges for the development of next-generation optoelectronic materials in the emerging field of bionic vision are discussed.

Surface defect detection competition with a bio-inspired vision sensor.

This paper reports the background and results of the Surface Defect Detection Competition with Bio-inspired Vision Sensor, as well as summarizes the champion solutions, current challenges and future directions.

DTFS-eHarris: A High Accuracy Asynchronous Corner Detector for Event Cameras in Complex Scenes

The event camera, a new bio-inspired vision sensor with low latency and high temporal resolution, has brought great potential and demonstrated a promising application in machine vision and artificial intelligence. Corner detection is a key step of object motion estimation and tracking. However, most existing event-based corner detectors, such as G-eHarris and Arc*, lead to a huge number of redundant or wrong corners, and cannot strike a balance between the accuracy and real-time performance, especially in complex scenes with high texture that require higher computational costs. To address these issues, we propose an asynchronous corner detection method: a double threshold filter with Sigmoid eHarris (DTFS-eHarris) and an asynchronous corner tracker. The main contributions are that a double threshold filter is designed to reduce the redundant events and the improved Sigmoid function is utilized to represent the Surface of Active Events (Sigmoid*-SAE). We selected four scenes—shapes, dynamic, poster and boxes—from the public event camera dataset DAVIS240C to compare with the existing state-of-the-art hybrid method; our method has shown more than a 10% reduction in false positive rate and a 5% and 20% improvement in accuracy and throughput, respectively. The evaluations indicate that DTFS-eHarris shows a significant improvement, especially in complex scenes. Thus, it is anticipated to enhance the real-time performance and feature detection accuracy for future robotic applications.

Modality Translation and Fusion for event-based semantic segmentation

Semantic segmentation is an important part of scene understanding in autonomous systems. Accurate segmentation results help the autonomous system better understand the surrounding environment, which improve the task success rate and enhance the system reliability. However, due to the hardware limitations of frame-based cameras, semantic segmentation based on RGB images under low-light conditions is still a difficult problem. The event camera, an emerging bio-inspired vision sensor, tries to fill these gaps with its high dynamic range. Most previous works resort to cross-modality knowledge distillation from a pretrained image-based teacher network to train an event-based network for semantic segmentation. However, a direct knowledge distillation is inadequate because of the different input modalities in the two networks. Utilizing image features to supervise event features cannot make full use of the knowledge in the teacher network. Thus, we propose a Modality Translation and Fusion (MTF) framework to distill diverse cross-modality knowledge. Specifically, we first develop a Modality Translation (MT) module to convert events into image modality. With different input modalities, two feature extractors are built to learn diverse and complementary knowledge from the teacher network. Then, to better utilize the knowledge, we propose a Residual-based Coordinate Attention Fusion (RCAF) module to fuse the multi-scale features from different modalities. Finally, extensive experiments show that our MTF is superior to state-of-the-art (SOTA) approaches.

Asynchronous Spatiotemporal Spike Metric for Event Cameras.

Event cameras as bioinspired vision sensors have shown great advantages in high dynamic range and high temporal resolution in vision tasks. Asynchronous spikes from event cameras can be depicted using the marked spatiotemporal point processes (MSTPPs). However, how to measure the distance between asynchronous spikes in the MSTPPs still remains an open issue. To address this problem, we propose a general asynchronous spatiotemporal spike metric considering both spatiotemporal structural properties and polarity attributes for event cameras. Technically, the conditional probability density function is first introduced to describe the spatiotemporal distribution and polarity prior in the MSTPPs. Besides, a spatiotemporal Gaussian kernel is defined to capture the spatiotemporal structure, which transforms discrete spikes into the continuous function in a reproducing kernel Hilbert space (RKHS). Finally, the distance between asynchronous spikes can be quantified by the inner product in the RKHS. The experimental results demonstrate that the proposed approach outperforms the state-of-the-art methods and achieves significant improvement in computational efficiency. Especially, it is able to better depict the changes involving spatiotemporal structural properties and polarity attributes.

Improving the Classification Accuracy in Label-Free Flow Cytometry Using Event-Based Vision and Simple Logistic Regression

Event-based cameras are novel bio-inspired vision sensors that do not follow the mechanism of traditional frame-based cameras. In the field of data acquisition, the replacement of CMOS cameras with event-based cameras has proved to enhance the accuracy of machine learning methods in situations where critical lighting conditions and rapid dynamics are paramount. In this paper, we investigate for the first time the use of extreme learning machines on data coming from event-based cameras in the context of flow cytometry. Except for the image sensor, the experimental setup is similar to a setup we used in (Lugnan et al., 2020) where we showed that a simple linear classifier can achieve around 10% error rate on background-subtracted cell frames. Here, we show that the the error rate of this simple imaging flow cytometer could be decreased to less than 2% just by making use of the capabilities of an event camera. Moreover, additional benefits like more sensitivity and efficient memory usage are gained. Finally, we suggest further possible improvements to the experimental setup used to record events from flowing micro-particles allowing for more accurate and stable cellx sorting.

CFSR: Coarse-to-Fine High-Speed Motion Scene Reconstruction with Region-Adaptive-Based Spike Distinction

As a novel bio-inspired vision sensor, spike cameras offer significant advantages over conventional cameras with a fixed low sampling rate, recording fast-moving scenes by firing a continuous stream of spikes. Reconstruction methods including Texture from ISI (TFI), Texture from Playback (TFP), and Texture from Adaptive threshold (TFA) produce undesirable noise or motion blur. A spiking neural model distinguishes the dynamic and static spikes before reconstruction, but the reconstruction of motion details is still unsatisfactory even with the advanced TFA method. To address this issue, we propose a coarse-to-fine high-speed motion scene reconstruction (CFSR) method with a region-adaptive-based spike distinction (RASE) framework to reconstruct the full texture of natural scenes from the spike data. We utilize the spike distribution of dynamic and static regions to propose the RASE to distinguish the spikes of different moments. After distinction, the TFI, TFP, and patch matching are exploited for image reconstruction in different regions, respectively, which does not introduce unexpected noise or motion blur. Experimental results on the PKU-SPIKE-RECON dataset demonstrate that our CFSR method outperforms the state-of-the-art approaches in terms of objective and subjective quality.

Learning Super-Resolution Reconstruction for High Temporal Resolution Spike Stream

Spike camera is a new type of bio-inspired vision sensor, each pixel of which perceives the brightness of the scene independently, and finally outputs 3-dimensional spatiotemporal spike streams. To bridge the spike camera and traditional frame-based vision, there is some works to reconstruct spike streams into regular images. However, the low spatial resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$400\times 250$ </tex-math></inline-formula> ) of the spike camera limits the quality of the reconstructed images. Thus, it is meaningful to explore a super-resolution reconstruction for spike streams. In this paper, we propose an end-to-end network to reconstruct high-resolution images from low-resolution spike streams. To utilize more spatiotemporal features of spike streams, our network adopts a multi-level features learning mechanism, including intra-stream feature extraction by spike encoder, inter-stream dependencies extraction based on optical flow module, and joint features learning via spike-based iterative projection. Experimental results demonstrate that our network is superior to the combination of state-of-the-art intensity image reconstruction methods and super-resolution networks on simulated and real datasets.

Understanding Human Reactions Looking at Facial Microexpressions With an Event Camera

With the establishment of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Industry 4.0</i> , machines are now required to interact with workers. By observing biometrics they can assess if humans are authorized, or mentally and physically fit to work. Understanding body language, makes human–machine interaction more natural, secure, and effective. Nonetheless, traditional cameras have limitations; low frame rate and dynamic range hinder a comprehensive human understanding. This poses a challenge, since faces undergo frequent instantaneous microexpressions. In addition, this is privacy-sensitive information that must be protected. We propose to model expressions with event cameras, bio-inspired vision sensors that have found application within the Industry 4.0 scope. They capture motion at millisecond rates and work under challenging conditions like low illumination and highly dynamic scenes. Such cameras are also privacy-preserving, making them extremely interesting for industry. We show that using event cameras, we can understand human reactions by only observing facial expressions. Comparison with red-green-blue (RGB)-based modeling demonstrates improved effectiveness and robustness.

Self-Supervised Intensity-Event Stereo Matching

Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes in microsecond accuracy with high dynamic range and low power consumption. Despite these advantages, event cameras cannot be directly applied to computational imaging tasks due to the inability to obtain high-quality intensity and events simultaneously. This paper aims to connect a standalone event camera and a modern intensity camera so that applications can take advantage of both sensors. We establish this connection through a multi-modal stereo matching task. We first convert events to a reconstructed image and extend the existing stereo networks to this multi-modality condition. We propose a self-supervised method to train the multi-modal stereo network without using ground truth disparity data. The structure loss calculated on image gradients is used to enable self-supervised learning on such multi-modal data. Exploiting the internal stereo constraint between views with different modalities, we introduce general stereo loss functions, including disparity cross-consistency loss and internal disparity loss, leading to improved performance and robustness compared to existing approaches. Our experiments demonstrate the effectiveness of the proposed method, especially the proposed general stereo loss functions, on both synthetic and real datasets. Finally, we shed light on employing the aligned events and intensity images in downstream tasks, e.g., video interpolation application.

REVIO: Range- and Event-Based Visual-Inertial Odometry for Bio-Inspired Sensors.

Visual-inertial odometry is critical for Unmanned Aerial Vehicles (UAVs) and robotics. However, there are problems of motion drift and motion blur in sharp brightness changes and fast-motion scenes. It may cause the degradation of image quality, which leads to poor location. Event cameras are bio-inspired vision sensors that offer significant advantages in high-dynamic scenes. Leveraging this property, this paper presents a new range and event-based visual-inertial odometry (REVIO). Firstly, we propose an event-based visual-inertial odometry (EVIO) using sliding window nonlinear optimization. Secondly, REVIO is developed on the basis of EVIO, which fuses events and distances to obtain clear event images and improves the accuracy of position estimation by constructing additional range constraints. Finally, the EVIO and REVIO are tested in three experiments-dataset, handheld and flight-to evaluate the localization performance. The error of REVIO can be reduced by nearly 29% compared with EVIO in the handheld experiment and almost 28% compared with VINS-Mono in the flight experiment, which demonstrates the higher accuracy of REVIO in some fast-motion and high-dynamic scenes.

Neuromorphic Vision-Based Fall Localization in Event Streams With Temporal-Spatial Attention Weighted Network.

Falling down is a serious problem for health and has become one of the major etiologies of accidental death for the elderly living alone. In recent years, many efforts have been paid to fall recognition based on wearable sensors or standard vision sensors. However, the prior methods have the risk of privacy leaks, and almost all these methods are based on video clips, which cannot localize where the falls occurred in long videos. For these reasons, in this article, the bioinspired vision sensor-based falls temporal localization framework is proposed. The bioinspired vision sensors, such as dynamic and active-pixel vision sensor (DAVIS) camera applied in this work responds to pixels' brightness change, and each pixel works independently and asynchronously compared to the standard vision sensors. This property makes it have a very high dynamic range and privacy preserving. First, to better represent event data, compared with the typical constant temporal window mechanism, an adaptive temporal window conversion mechanism is developed. The temporal localization framework follows a proven proposal and classification paradigm. Second, for the high-efficient and recall proposal generation, different from the traditional sliding window scheme, the event temporal density as the actionness score is set and the 1D-watershed algorithm to generate proposals is applied. In addition, we combine the temporal and spatial attention mechanism with our feature extraction network to temporally model the falls. Finally, to evaluate the performance of our framework, 30 volunteers are recruited to join the simulated fall experiments. According to the results of experiments, our framework can realize precise falls temporal localization and achieve the state-of-the-art performance.

Image De-occlusion via Event-enhanced Multi-modal Fusion Hybrid Network

Seeing through dense occlusions and reconstructing scene images is an important but challenging task. Traditional frame-based image de-occlusion methods may lead to fatal errors when facing extremely dense occlusions due to the lack of valid information available from the limited input occluded frames. Event cameras are bio-inspired vision sensors that record the brightness changes at each pixel asynchronously with high temporal resolution. However, synthesizing images solely from event streams is ill-posed since only the brightness changes are recorded in the event stream, and the initial brightness is unknown. In this paper, we propose an event-enhanced multi-modal fusion hybrid network for image de-occlusion, which uses event streams to provide complete scene information and frames to provide color and texture information. An event stream encoder based on the spiking neural network (SNN) is proposed to encode and denoise the event stream efficiently. A comparison loss is proposed to generate clearer results. Experimental results on a large-scale event-based and frame-based image de-occlusion dataset demonstrate that our proposed method achieves state-of-the-art performance.

NeuroGrasp: Multimodal Neural Network With Euler Region Regression for Neuromorphic Vision-Based Grasp Pose Estimation

Grasp pose estimation is a crucial procedure in robotic manipulation. Most of the current robot grasp manipulation systems are built on frame-based cameras like RGB-D cameras. However, the traditional frame-based grasp pose estimation methods have encountered challenges in scenarios such as low dynamic range and low power consumption. In this work, a neuromorphic vision sensor (DAVIS) is introduced to the field of robotic grasp. DAVIS is an event-based bio-inspired vision sensor that records asynchronous streams of local pixel-level light intensity changes, called events. The strengths of DAVIS are it can provide high temporal resolution, high dynamic range, low power consumption, and no motion blur. We construct a neuromorphic vision-based robotic grasp dataset with 154 moving objects, named NeuroGrasp, which is the first RGB-Event multi-modality grasp dataset (to the best of our knowledge). This dataset records both RGB frames and the corresponding event streams, providing frame data with rich color and texture information and event streams with high temporal resolution and high dynamic range. Based on the NeuroGrasp dataset, we further develop a multi-modal neural network with a specific Euler-Region-Regression sub-Network (ERRN) to perform grasp pose estimation. Combining with frame-based and event-based vision, the proposed method achieves better performance than the method that only takes RGB frames or event streams as input on the NeuroGrasp dataset.