Rethinking Benchmarks for Neuromorphic Learning Algorithms

Abstract

We rely on benchmarking datasets to monitor the research progress. However, recent studies have cast doubts on the effectiveness of current neuromorphic benchmarking datasets; and the debate remains largely unsettled. In this paper, we assess the richness and usefulness of temporal information embedded in these benchmarking datasets for SNN decision making. To this end, we propose a segregated spatio-temporal learning framework that allows us to selectively control the information flow along both spatial and temporal directions during feedforward and backward propagation. Leveraging on this framework, we conduct a comprehensive study on seven widely used neuromorphic audio and vision datasets. Our findings are threefold. First, the existing neuromorphic benchmarks only make limited contributions in highlighting the temporal processing capability of spiking neurons. Second, the temporal credit assignment is redundant for tasks that only require short-range temporal dependency. Third, we recommend the neuromorphic research community to develop novel benchmarks that require both short-range and long-range temporal dependencies. Such appropriate benchmark datasets would be helpful in guiding the development of powerful SNN-based learning algorithms and computational models.