Silicon Retina Research Articles

Tau-Cell-Based Analog Silicon Retina With Spatio- Temporal Filtering and Contrast Gain Control.

Developing precise artificial retinas is crucial because they hold the potential to restore vision, improve visual prosthetics, and enhance computer vision systems. Emulating the luminance and contrast adaption features of the retina is essential to improve visual perception and efficiency to provide an environment realistic representation to the user. In this article, we introduce an artificial retina model that leverages its potent adaptation to luminance and contrast to enhance vision sensing and information processing. The model has the ability to achieve the realization of both tonic and phasic cells in the simplest manner. We have implemented the retina model using 0.18 μm process technology and validated the accuracy of the hardware implementation through circuit simulation that closely matches the software retina model. Additionally, we have characterized a single pixel fabricated using the same 0.18 μm process. This pixel demonstrates an 87.7-% ratio of variance with the temporal software model and operates with a power consumption of 369 nW.

Silicon retina: Focusing on motion studies for detecting the x-axes and y-axes

<p>The purpose of this report is to explore how to build a circuit that works as a human retina, which can detect motion. The whole project is divided into five steps, which include researching, simulating, building a cell on the breadboard and testing, presenting a cell on Printed Circuit Board and testing, printing more cells, and connecting all the cells. Through each step of the execution, the output signals from the <em>x</em>-axis and <em>y</em>-axis become more and more accurate, in other words, the speed measured by the entire retinal system becomes more and more accurate. When connecting the photoreceptor as the input, different motion types, which include moving fast or slow, moving horizontally or vertically, and moving upward at an angle or downward at an angle, and the results are also basically following the analysis and the expected result read from the books or journals. The errors are mainly caused by the different light conditions in the environment.</p>

Modelling of FinFET based ON bipolar cell of silicon retina

—In this paper, we proposed a SOI FinFET based ON bipolar cell of silicon retina which plays an important role in neuromorphic applications like differential motion detection. Low power consumption along with biological plausibility is one of the crucial parameters of any neuromorphic retinal circuit as in the brain all biological systems consisting of billions of neurons consume power in terms of a few μwatts. First, the output of linkage cell between inner and outer plexiform layer i.e. bipolar cell is reconstructed and verified using developed ionic current based mathematical model which is established considering Hodgkin-Huxley model as the base model, and then biological behavior of the ON bipolar cell is mimicked through circuit using 180 nm, 90 nm and 45 nm Metal Oxide Semiconductor Field Effect Transistor (MOSFET). The power consumption in each case is obtained to be approximately 18.5 nW, 10.5 nW and 1 nW which is much larger than the power consumed by biological bipolar cell. Moreover, the same circuit is implemented with 32 nm Silicon on Insulator Fin Field Effect Transistor (SOI FinFET) device. It is observed that SOI FinFET based ON bipolar cell consumes 17.5 pW power which is less in comparison to MOSFET based bipolar cell. The output of bipolar cell is matched with biological nature of the waveform in each case.

FinFET based photoreceptor for silicon retina to reduce power consumption

In this paper, we propose a SOI FinFET device based photoreceptor for silicon retina to reduce power consumption. Nowadays, in the field of neuromorphic engineering the main aim of the researcher is to implement a biological nervous system based on silicon devices. Major performance metrics which influence the design of neuromorphic circuits along with biological plausibility are compactness, low power consumption, scalability and computational complexity. Moreover, to process the information biologically billions of neurons consume only a few watts of power. However, present design of silicon photoreceptor consumes power much larger than actual biological photoreceptor. Hence to achieve low power consumption we are using nanometer SOI FinFET device technology. In this paper, output of the outer retinal cell, namely photoreceptors, is reconstructed and verified with the help of developed ionic current based mathematical model of the retina. The model contains non-linear differential equations based on the fundamental framework of the Hodgkin-Huxley model. In order to closely match the biological response, the selection of parameters of membrane voltage and other membrane dependencies through compartments are done using MATLAB simulation. After mathematical verification, the biological behavior of photoreceptor cell of retinal system is mimicked through circuit using 180 nm Metal Oxide Semiconductor Field Effect Transistor (MOSFET) and 32 nm Silicon on Insulator Fin Field Effect Transistor (SOI FinFET) device. It is observed that SOI FinFET based photoreceptor for silicon retina circuit consumes less power in picowatts matching the power consumption of biological retinal system, as compared with MOSFET based circuits.

NeuroIV: Neuromorphic Vision Meets Intelligent Vehicle Towards Safe Driving With a New Database and Baseline Evaluations

Neuromorphic vision sensors such as the Dynamic and Active-pixel Vision Sensor (DAVIS) using silicon retina are inspired by biological vision, they generate streams of asynchronous events to indicate local log-intensity brightness changes. Their properties of high temporal resolution, low-bandwidth, lightweight computation, and low-latency make them a good fit for many applications of motion perception in the intelligent vehicle. However, as a younger and smaller research field compared to classical computer vision, neuromorphic vision is rarely connected with the intelligent vehicle. For this purpose, we present three novel datasets recorded with DAVIS sensors and depth sensor for the distracted driving research and focus on driver drowsiness detection, driver gaze-zone recognition, and driver hand-gesture recognition. To facilitate the comparison with classical computer vision, we record the RGB, depth and infrared data with a depth sensor simultaneously. The total volume of this dataset has 27360 samples. To unlock the potential of neuromorphic vision on the intelligent vehicle, we utilize three popular event-encoding methods to convert asynchronous event slices to event-frames and adapt state-of-the-art convolutional architectures to extensively evaluate their performances on this dataset. Together with qualitative and quantitative results, this work provides a new database and baseline evaluations named NeuroIV in cross-cutting areas of neuromorphic vision and intelligent vehicle.

FinFET Based Photoreceptor for Silicon Retina to Reduce Power Consumption

FinFET Based Photoreceptor for Silicon Retina to Reduce Power Consumption

Unsupervised Adaptive Multi-Object Tracking-by-Clustering Algorithm With a Bio-Inspired System

A problem of current interest is how to emulate nature by acquiring information in a neuromorphic-like fashion; namely, by using configurable hardware and electronic systems to emulate the information gathering and processing strategies of biological systems. In this paper, we introduce BioCAMSHIFT, an algorithm for a bio-inspired system that acquires information via a neuromorphic process and uses it to track multiple objects. The system consists of a silicon retina that simulates the behavior of the human eye together with a communication system that uses an Address-Event Representation protocol to transmit information in a way analogous to that of biological neural systems. An unsupervised procedure, based on the CAMSHIFT algorithm, is then used for multi-object tracking. It takes advantage of the retina’s high event rate to adapt to the changing sizes of the objects in its field of view. The proposed system has been experimentally validated using a data set from Freeway 210 in Pasadena, California, demonstrating a significantly better improvement in terms of multi-vehicle detection and tracking performance over the current state of the art.

Bionic eye: An iconic innovation

The bionic eye is a visual prosthesis that restores the vision fully or partly in blind people suffering from retinitis pigmentosa and macular degenerations. Many bionic eyes have completed human clinical trials and become functional but still need newer technical approaches to make it available to all people. A bionic eye illuminates the dark world of blind people. To date, many bionic eyes were developed like Argus II Retinal Prosthesis System, Artificial silicon retina and many more, whereas Gennaris Bionic vision System is the recent advanced bionic eye. In our paper, we have covered the basic structure and working of the bionic eye, projects on the bionic eye to date, vision through the bionic eye, implants and restoration of sight, limitations, and the future of the bionic eye.

Neuromorphic vision sensors: Principle, progress and perspectives

Conventional frame-based image sensors suffer greatly from high energy consumption and latency. Mimicking neurobiological structures and functionalities of the retina provides a promising way to build a neuromorphic vision sensor with highly efficient image processing. In this review article, we will start with a brief introduction to explain the working mechanism and the challenges of conventional frame-based image sensors, and introduce the structure and functions of biological retina. In the main section, we will overview recent developments in neuromorphic vision sensors, including the silicon retina based on conventional Si CMOS digital technologies, and the neuromorphic vision sensors with the implementation of emerging devices. Finally, we will provide a brief outline of the prospects and outlook for the development of this field.

A Multi-Modal, Silicon Retina Technique for Detecting the Presence of Reflective and Transparent Barriers

There is currently great interest in enhancing the ability of aerial robots to navigate indoors. Navigating a building under various lighting and environmental conditions would have application in disaster response, infrastructure inspections, as well as a wide variety of commercial applications. In order to achieve this goal, one common feature of indoor environments that must be addressed is the detection of transparent/reflective barriers. Transparent/reflective barriers as they pertain to structures generally take the form of a window, office divider, or storefront. Human tele-operators of aerial robots in environments such as malls, airports, office buildings, and museums that feature transparent barriers will need some means to enhance their situational awareness so they can recognize the presence of transparent/reflective barriers, distinguish between the two, and have some idea of the distance and pose relative to the transparent/reflective barrier. The ability to detect and localize transparent barriers will also be important for autonomous navigation. The focus of this work is to develop a multi-modal sensing solution that can successfully identify transparent/reflective barriers, distinguish between the two, and provide information on pose and distance to the barrier at time-scales exceeding human response in order to facilitate navigation of indoor spaces. The sensing solution relies on using an imager to measure the differences in the interactions of actively visible light illumination of transparent/reflective barriers. A silicon retina event-driven imager is used in this work to provide a path to obtaining information on transparent/reflective barriers at high speeds, while requiring very little communications bandwidth.

Bandwidth Modeling of Silicon Retinas for Next Generation Visual Sensor Networks.

Silicon retinas, also known as Dynamic Vision Sensors (DVS) or event-based visual sensors, have shown great advantages in terms of low power consumption, low bandwidth, wide dynamic range and very high temporal resolution. Owing to such advantages as compared to conventional vision sensors, DVS devices are gaining more and more attention in various applications such as drone surveillance, robotics, high-speed motion photography, etc. The output of such sensors is a sequence of events rather than a series of frames as for classical cameras. Estimating the data rate of the stream of events associated with such sensors is needed for the appropriate design of transmission systems involving such sensors. In this work, we propose to consider information about the scene content and sensor speed to support such estimation, and we identify suitable metrics to quantify the complexity of the scene for this purpose. According to the results of this study, the event rate shows an exponential relationship with the metric associated with the complexity of the scene and linear relationships with the speed of the sensor. Based on these results, we propose a two-parameter model for the dependency of the event rate on scene complexity and sensor speed. The model achieves a prediction accuracy of approximately 88.4% for the outdoor environment along with the overall prediction performance of approximately 84%.

A Spiking Neural Network Model of Depth from Defocus for Event-based Neuromorphic Vision

Depth from defocus is an important mechanism that enables vision systems to perceive depth. While machine vision has developed several algorithms to estimate depth from the amount of defocus present at the focal plane, existing techniques are slow, energy demanding and mainly relying on numerous acquisitions and massive amounts of filtering operations on the pixels’ absolute luminance value. Recent advances in neuromorphic engineering allow an alternative to this problem, with the use of event-based silicon retinas and neural processing devices inspired by the organizing principles of the brain. In this paper, we present a low power, compact and computationally inexpensive setup to estimate depth in a 3D scene in real time at high rates that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. Exploiting the high temporal resolution of the event-based silicon retina, we are able to extract depth at 100 Hz for a power budget lower than a 200 mW (10 mW for the camera, 90 mW for the liquid lens and ~100 mW for the computation). We validate the model with experimental results, highlighting features that are consistent with both computational neuroscience and recent findings in the retina physiology. We demonstrate its efficiency with a prototype of a neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological depth from defocus experiments reported in the literature.

Toward Brain-Inspired Learning With the Neuromorphic Snake-Like Robot and the Neurorobotic Platform

Neurorobotic mimics the structural and functional principles of living creature systems. Modeling a single system by robotic hardware and software has existed for decades. However, an integrated toolset studying the interaction of all systems has not been demonstrated yet. We present a hybrid neuromorphic computing paradigm to bridge this gap by combining the neurorobotics platform (NRP) with the neuromorphic snake-like robot (NeuroSnake). This paradigm encompasses the virtual models, neuromorphic sensing and computing capabilities, and physical bio-inspired bodies, with which an experimenter can design and execute both in-silico and in-vivo robotic experimentation easily. The NRP is a public Web-based platform for easily testing brain models with virtual bodies and environments. The NeuroSnake is a bio-inspired robot equipped with a silico-retina sensor and neuromorphic computer for power-efficiency applications. We illustrate the efficiencies of our paradigm with an easy designing of a visual pursuit experiment in the NRP. We study two automatic behavior learning tasks which are further integrated into a complex task of semi-autonomous pole climbing. The result shows that robots could build new learning rules in a less explicit manner inspired by living creatures. Our method gives an alternative way to efficiently develop complex behavior control of the ro As spiking neural network is a bio-inspired neural network and the NeuroSnake robot is equipped with a spike-based silicon retina camera, the control system can be easily implemented via spiking neurons simulated on neuromorphic hardware, such as SpiNNaker.bot.

Event-based Sensing for Space Situational Awareness

A revolutionary type of imaging device, known as a silicon retina or event-based sensor, has recently been developed and is gaining in popularity in the field of artificial vision systems. These devices are inspired by a biological retina and operate in a significantly different way to traditional CCD-based imaging sensors. While a CCD produces frames of pixel intensities, an event-based sensor produces a continuous stream of events, each of which is generated when a pixel detects a change in log light intensity. These pixels operate asynchronously and independently, producing an event-based output with high temporal resolution. There are also no fixed exposure times, allowing these devices to offer a very high dynamic range independently for each pixel. Additionally, these devices offer high-speed, low power operation and a sparse spatio-temporal output. As a consequence, the data from these sensors must be interpreted in a significantly different way to traditional imaging sensors and this paper explores the advantages this technology provides for space imaging. The applicability and capabilities of event-based sensors for SSA applications are demonstrated through telescope field trials. Trial results have confirmed that the devices are capable of observing resident space objects from LEO through to GEO orbital regimes. Significantly, observations of RSOs were made during both day-time and night-time (terminator) conditions without modification to the camera or optics. The event based sensor’s ability to image stars and satellites during day-time hours offers a dramatic capability increase for terrestrial optical sensors. This paper shows the field testing and validation of two different architectures of event-based imaging sensors. An event-based sensor’s asynchronous output has an intrinsically low data-rate. In addition to low-bandwidth communications requirements, the low weight, low-power and high-speed make them ideally suitable to meeting the demanding challenges required by space-based SSA systems. Results from these experiments and the systems developed highlight the applicability of event-based sensors to ground and space-based SSA tasks.

Front and Back Illuminated Dynamic and Active Pixel Vision Sensors Comparison

Back side illumination has become standard image sensor technology owing to its superior quantum efficiency and fill factor. A direct comparison of front and back side illumination (FSI and BSI) used in event-based dynamic and active pixel vision sensors (DAVIS) is interesting because of the potential of BSI to greatly increase the small 20% fill factor of these complex pixels. This brief compares identically designed front and back illuminated DAVIS silicon retina vision sensors. They are compared in term of quantum efficiency (QE), leak activity and modulation transfer function (MTF). The BSI DAVIS achieves a peak QE of 93% compared with the FSI DAVIS, peak QE of 24%, but reduced MTF, due to pixel crosstalk and parasitic photocurrent. Significant “leak events” in the BSI DAVIS limit its use to controlled illumination scenarios without very bright light sources. Effects of parasitic photocurrent and modulation transfer functions with and without IR cut filters are also reported.

Neuromorphic sensorimotor adaptation for robotic mobile manipulation: From sensing to behaviour

We propose a neuromorphic approach to perception, reasoning and motor control using Spiking Neural Networks in mobile robotics. We demonstrate this by using a mobile robotic manipulator solving a pick-and-place task. All sensory data is provided by spike-based silicon retina cameras - eDVS (embedded Dynamic Vision Sensor) - and all reasoning and motor control is implemented in Spiking Neural Networks. For the given scenario, the robot is capable of detecting a sequence of objects blinking at different frequencies, finding one object that is not in the right place of the sequence, picking up this object and moving it to its correct position. Such a scenario demonstrates how to build large-scale networks solving a high-level cognitive task by combining several smaller networks responsible for low-level tasks. Importantly, here we focus only on generating a neural network that is capable of performing the task. This will be the basis of future work using neural network learning algorithms to improve task performance. The long-term goal is to learn sophisticated behaviours by experience while at the same time being able to introduce expert knowledge for intermediate tasks that can be used to initialize the network or to speed up the learning process.

A Noise Filtering Algorithm for Event-Based Asynchronous Change Detection Image Sensors on TrueNorth and Its Implementation on TrueNorth.

Asynchronous event-based sensors, or “silicon retinae,” are a new class of vision sensors inspired by biological vision systems. The output of these sensors often contains a significant number of noise events along with the signal. Filtering these noise events is a common preprocessing step before using the data for tasks such as tracking and classification. This paper presents a novel spiking neural network-based approach to filtering noise events from data captured by an Asynchronous Time-based Image Sensor on a neuromorphic processor, the IBM TrueNorth Neurosynaptic System. The significant contribution of this work is that it demonstrates our proposed filtering algorithm outperforms the traditional nearest neighbor noise filter in achieving higher signal to noise ratio (~10 dB higher) and retaining the events related to signal (~3X more). In addition, for our envisioned application of object tracking and classification under some parameter settings, it can also generate some of the missing events in the spatial neighborhood of the signal for all classes of moving objects in the data which are unattainable using the nearest neighbor filter.

Low-Latency Line Tracking Using Event-Based Dynamic Vision Sensors.

In order to safely navigate and orient in their local surroundings autonomous systems need to rapidly extract and persistently track visual features from the environment. While there are many algorithms tackling those tasks for traditional frame-based cameras, these have to deal with the fact that conventional cameras sample their environment with a fixed frequency. Most prominently, the same features have to be found in consecutive frames and corresponding features then need to be matched using elaborate techniques as any information between the two frames is lost. We introduce a novel method to detect and track line structures in data streams of event-based silicon retinae [also known as dynamic vision sensors (DVS)]. In contrast to conventional cameras, these biologically inspired sensors generate a quasicontinuous stream of vision information analogous to the information stream created by the ganglion cells in mammal retinae. All pixels of DVS operate asynchronously without a periodic sampling rate and emit a so-called DVS address event as soon as they perceive a luminance change exceeding an adjustable threshold. We use the high temporal resolution achieved by the DVS to track features continuously through time instead of only at fixed points in time. The focus of this work lies on tracking lines in a mostly static environment which is observed by a moving camera, a typical setting in mobile robotics. Since DVS events are mostly generated at object boundaries and edges which in man-made environments often form lines they were chosen as feature to track. Our method is based on detecting planes of DVS address events in x-y-t-space and tracing these planes through time. It is robust against noise and runs in real time on a standard computer, hence it is suitable for low latency robotics. The efficacy and performance are evaluated on real-world data sets which show artificial structures in an office-building using event data for tracking and frame data for ground-truth estimation from a DAVIS240C sensor.

Feature Representations for Neuromorphic Audio Spike Streams.

Event-driven neuromorphic spiking sensors such as the silicon retina and the silicon cochlea encode the external sensory stimuli as asynchronous streams of spikes across different channels or pixels. Combining state-of-art deep neural networks with the asynchronous outputs of these sensors has produced encouraging results on some datasets but remains challenging. While the lack of effective spiking networks to process the spike streams is one reason, the other reason is that the pre-processing methods required to convert the spike streams to frame-based features needed for the deep networks still require further investigation. This work investigates the effectiveness of synchronous and asynchronous frame-based features generated using spike count and constant event binning in combination with the use of a recurrent neural network for solving a classification task using N-TIDIGITS18 dataset. This spike-based dataset consists of recordings from the Dynamic Audio Sensor, a spiking silicon cochlea sensor, in response to the TIDIGITS audio dataset. We also propose a new pre-processing method which applies an exponential kernel on the output cochlea spikes so that the interspike timing information is better preserved. The results from the N-TIDIGITS18 dataset show that the exponential features perform better than the spike count features, with over 91% accuracy on the digit classification task. This accuracy corresponds to an improvement of at least 2.5% over the use of spike count features, establishing a new state of the art for this dataset.

CMOS Vision Sensors: Embedding Computer Vision at Imaging Front-Ends

CMOS Image Sensors (CIS) are key for imaging technologies. These chips are conceived for capturing optical scenes focused on their surface, and for delivering electrical images, commonly in digital format. CISs may incorporate intelligence; however, their smartness basically concerns calibration, error correction and other similar tasks. The term CVISs (CMOS VIsion Sensors) defines other class of sensor front-ends which are aimed at performing vision tasks right at the focal plane. They have been running under names such as computational image sensors, vision sensors and silicon retinas, among others.