Inspired by nature: timescale-free and grid-free event-based computing with spiking neural networks

Abstract

Computer vision is enjoying huge success in visual processing applications such as facial recognition, object identification, and navigation. Most of these studies work with traditional cameras which produce frames at predetermined fixed time intervals. Real life visual stimuli are, however, generated when changes occur in the environment and are irregular in timing. Biological visual neural systems operate on these changes and are hence free from any fixed timescales that are related to the timing of events in visual input.Inspired by biological systems, neuromorphic devices provide a new way to record visual data. These devices typically have parallel arrays of sensors which operate asynchronously. They have particular potential for robotics due to their low latency, efficient use of bandwidth and low power requirements. There are a variety of neuromorphic devices for detecting different sensory information; this thesis focuses on using the Dynamic Vision Sensor (DVS) for visual data collection.Event-based sensory inputs are generated on demand as changes happen in the environment. There are no systematic timescales in these activities and the asynchronous nature of the sensors adds to the irregularity of time intervals between events, making event-based data timescale-free. Although the array of sensors are arranged as a grid in vision sensors generally, events in the real world exist in continuous space. Biological systems are not restricted to grid-based sampling, and it is an open question whether event-based data could similarly take advantage of grid-free processing algorithms. To study visual data in a way which is timescale-free and grid-free, which is  fundamentally different from traditional video data sampled at fixed time intervals which are dense and rigid in space, requires conceptual viewpoints and methods of computation which are not typically employed in existing studies.Bio-inspired computing involves computational components that mimic or at least take inspiration from how nature works. This fusion of engineering and biology often provides insights into complex computational problems. Artificial neural networks, a computing paradigm that is inspired by how our brains work, have been studied widely with visual data. This thesis uses a type of artificial neural network—event-based spiking neural networks—as the basic framework to process event-based visual data.Building upon spiking neural networks, this thesis introduces two methods that process event-based data with the principles of being timescale-free and grid-free. The first method preprocesses events as distributions of Gaussian shaped spatiotemporal volumes, and then introduces a new neuron model with time-delayed dendrites and dendritic and axonal computation as the main building blocks of the spiking neural network to perform long-term predictions. Gaussians are used for simplicity purposes. This Gaussian-based method is shown in this thesis to outperform a commonly used iterative prediction paradigm on DVS data.The second method involves a new concept for processing event-based data based on the “light cone” idea in physics. Starting from a given point in real space at a given time, a light cone is the set of points in spacetime reachable without exceeding the speed of light, and these points trace out spacetime trajectories called world lines. The light cone concept is applied to DVS data. As an object moves with respect to the DVS, the events generated are related by their speeds relative to the DVS. An observer can calculate possible world lines for each point but has no access to the correct one. The idea of a “motion cone” is introduced to refer to the distribution of possible world lines for an event. Motion cones provide a novel theory for the early stages of visual processing. Instead of spatial clustering, world lines produce a new representation determined by a speed-based clustering of events. A novel spiking neural network model with dendritic connections based on motion cones is proposed, with the ability predict future motion pattern in a long-term prediction.Freedom from timescales and fixed grid sizes are fundamental characteristics of neuromorphic event-based data but few algorithms to date exploit their potential. Focusing on the inter-event relationship in the continuous spatiotemporal volume can preserve these features during processing. This thesis presents two examples of incorporating the features of being timescale-free and grid-free into algorithm development and examines their performance on real world DVS data. These new concepts and models contribute to the neuromorphic computation field by providing new ways of thinking about event-based representations and their associated algorithms. They also have the potential to stimulate rethinking of representations in the early stages of an event-based vision system. To aid algorithm development, a benchmarking data set containing data ranging from simple environment changes collected from a stationary camera to complex environmentally rich navigation performed by mobile robots has been collated. Studies conducted in this thesis use examples from this benchmarking data set which is also made available to the public.