An End-to-end Computer Vision System Architecture

Abstract

To overcome the data movement bottleneck, near-sensor and in-sensor computing are becoming more and more popular. However, in the existing near-/in-sensor computing architectures for vision tasks, the effect of the image signal processing (ISP) pipeline, which is of great importance to the final vision performance [1], is always ignored. In this work, we propose a synthesized RAW image-based end-to-end computer vision paradigm, taking the effect of ISP pipeline into account. In the proposed approach, a generative adversarial network (GAN)-based tool is used to convert the fully processed color images to their corresponding RAW Bayer versions, generating the training data for end-to-end vision models. In the inference stage, RAW images from the sensor are directly fed to the end-to-end model, bypassing the entire ISP pipeline. Experimental results show that by training/tuning the CNN models using synthesized RAW images, it is possible to design an end-to-end (from RAW image to vision task) vision system that directly consumes RAW image data from the sensor with negligible vision performance degradation. By skipping the ISP pipeline, an image sensor can be directly integrated with the back-end vision processor without a complex image processor in the middle, making near-/in-sensor computing a practical approach.