An On-Chip Binary-Weight Convolution CMOS Image Sensor for Neural Networks

Abstract

A CMOS image sensor (CIS) that can perform on-chip binary convolution is presented. The CIS can greatly reduce memory usage and computational complexity by directly generating a feature map for a binary neural network. The pixel readout of the CIS is performed in the column-parallel fashion using incremental delta-sigma analog-to-digital converters (ADCs). The CIS operates in two different modes: convolution and normal modes. When the column ADC is working in the convolution mode, it works as a first-order delta-sigma ADC and generates convolved images using a binary kernel. In the normal operation mode, the ADC is switched to a second-order delta-sigma ADC with little hardware modification and used to capture high-quality images. To demonstrate the CIS architecture, a 192 × 128-pixel CIS, which occupies an active die area of 14.44 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , is fabricated in a 0.18 μm standard CMOS process. The performance of the CIS is evaluated through measurements and network simulations. In the normal operation mode, the CIS achieves a read noise of 14.79 e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-</sup> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">rms</sub> and a full-well capacity of 6,420 e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-</sup> with a resulting dynamic range of 53 dB. The power consumptions of the CIS are 49.2 and 52.5 mW during the normal and convolution modes, respectively.