Embedded neuromorphic vision systems

Abstract

The large body of research in perceptual computing has and will continue to enable many intriguing applications such as augmented reality, driver assistance, and personal analytics. Moreover, as wearable first person computing devices become increasingly popular, the demand for highly interactive perceptual computing applications will increase rapidly. Applications including first person assistance and analytics will be pervasive across retail, automotive, and medical domains. However, the computational requirements demanded by future perceptual computing applications will far exceed the capabilities of traditional vision algorithms that are executed on sequential CPUs and GPUs. Hardware accelerators are recognized as key to surpassing the limits of existing sequential architectures. In particular, brain inspired, or neuromorphic, vision accelerators have the potential to support computationally intensive perception algorithms on resource and power constrained devices.It is estimated that the human brain consumes a mere 20 watts of power as it perceives and comprehends the complex natural world. While the capability and efficiency of the human brain is indisputable, there are many perspectives on how to translate these efficiencies to algorithmic models and computational structures that are realizable on embedded platforms. Certainly, as advancements are made to neuromorphic vision models, identifying next generation compute architectures that exploit the inherent robustness of these models is critical to enabling compelling perceptual computing in future embedded devices. Unfortunately, executing perceptual computing applications on traditional CPU and GPU architectures will neither meet performance requirements nor the power constraints of future smartphones, smart watches, and wearable computing glasses. As such there has been increased interest in architecting and employing domain-specific accelerators that support a broad set of neuromorphic vision models. Accordingly, our research has focused on developing high performance neuromorphic accelerator architectures and System-on-Chip designs for resource and power constrained embedded platforms. While there is ongoing research on realizing brain-like hardware fabrics based on analog spiking networks, our work has mainly focused on realizing systems using digital accelerators. Attributed to the dataflow nature of neuromorphic algorithms, the need to implement complex and power consuming control-flow structures is reduced when mapping to custom hardware. Moreover, when appropriately mapped to streaming accelerators, costly off-chip memory accesses can be reduced significantly. The result of our work are several accelerators for image data fusion, bottom-up image saliency, scene GIST extraction, and cortical based object classification. These end-to-end pipelines have been architected with particular focus on maximizing performance per watt which is critical for integration into power constrained embedded platforms. Performance evaluations have shown that our accelerated neuromorphic vision systems exhibit several orders of energy efficiency over state-of-art machine vision implementations on multicore CPU and GPUs.In addition to exploring energy efficient architectures of neuromorphic accelerators, our research investigates ways of leveraging the inherent robustness of neuromorphic models to gain increased energy efficiency. We have found that by modulating the effort level and expected accuracy of object classification based on importance based ranking, overall accuracy of scene classification tasks can be maintained while reducing power consumption and off-chip memory bandwidth requirements. In recent work we employ accelerators in a framework that ranks the importance of regions-of-interests based on their saliency score. This ranking is subsequently used to modulate the degree of exhaustiveness that Gabor filters are applied at various stages in the HMAX classification model. While accuracy is only marginally decreased, overall power consumption is reduced by over 60%. Moreover, by utilizing a similar region-of-interest ranking technique the off-chip memory bandwidth required to provide data to several object classification accelerators is reduced significantly. Again, with minimal degradation of accuracy power consumption is reduced.The availability of low-power, high-speed and high-accuracy vision systems that detect and classify, will enable a variety of compelling surveillance, driver assistance, and person analytics systems. This talk highlights our experiences in architecting several embedded neuromorphic vision systems. Several design strategies and their subsequent implications on performance and energy efficiency are presented. Furthermore, future work that takes advantage of the robust nature of neuromorphic models in an approximate computing framework is presented.