Hardware-Compliant Compressive Image Sensor Architecture Based on Random Modulations and Permutations for Embedded Inference

Abstract

This work presents a compact CMOS Image Sensor (CIS) architecture enabling embedded object recognition facilitated by a dedicated end-of-column Compressive Sensing (CS), reducing on-chip memory needs. Our sensing scheme is based on a combination of random modulations and permutations leading to an implementation with very limited hardware impacts. It is designed to meet both theoretical ( i.e. , stable embedding, measurements incoherence) and practical requirements ( i.e. , silicon footprint, power consumption). The only additional hardware compared to a standard CIS architecture using first order incremental Sigma-Delta ( $\Sigma \Delta $ ) Analog to Digital Converter (ADC) are a pseudo-random data mixing circuit, an in- $\Sigma \Delta ~\pm 1$ modulator and a small Digital Signal Processor (DSP). On the algorithmic side, three variants are presented to perform the inference on compressed measurements with a tunable complexity ( i.e. , one-vs.-all SVM, hierarchical SVM and small ANN with 1-D max-pooling). An object recognition accuracy of $\simeq 98.8$ % is reached on the COIL database (COIL, 100 classes) using our dedicated Neural Network classifier. We stress that the signal-independent dimensionality reduction performed by our dedicated CS scheme (1/480 in $480 \times 640$ VGA resolution case) allows to dramatically reduce memory requirements mainly related to the remotely learned coefficients used for the inference stage.