A Fully Dynamic Multi-Mode CMOS Vision Sensor With Mixed-Signal Cooperative Motion Sensing and Object Segmentation for Adaptive Edge Computing

Abstract

This article presents a low-power multi-mode CMOS vision sensor with mixed-signal in-sensor computation capabilities targeting the next-generation wireless sensing applications. To support the always-on and scene-adaptive edge computing scenarios with low power and low bandwidth, the sensor is reconfigurable for three operation modes, namely: 1) motion sensing (MS); 2) object segmentation (OS); and 3) full imaging (FIM). A mixed-signal cooperative scheme of frame differencing (FD) and background subtraction (BS) is proposed to achieve high-accuracy MS with varying object sizes and speeds. The mixed-signal BS-based OS can minimize both object localizing and imaging efforts for object analysis upon motion triggering, while FIM enables complete scene recording for the identified object of interest. The complete CMOS vision sensor is implemented through reconfigurable and fully dynamic mixed-signal processing at both pixel and column levels cooperatively to achieve low power and compact area. Fabricated in a 0.18- ${ {\mu }}\text{m}$ CMOS, the 256 $ {\times }$ 216 chip prototype achieves the cooperative MS with only 2.36 ${\mu }\text{W}$ at 15 frames/s, when composed of 14 FD frames (147 nJ/frame) and 1 BS frame (302 nJ/frame). The OS mode consumes 1.44~2.04 $ {\mu }\text{J}$ /frame at 0%~100% object occupancy, linearly corresponding to 41%~16% power saving when compared with the conventional digital OS. The FIM mode operates with only 1.41 ${\mu }\text{J}$ /frame for complete scene recording. The achieved energy efficiencies for all operation modes compare favorably with the state of the art.