Refactoring ISP for High-Level Vision Tasks

Abstract

The image signal processing (ISP) pipeline, which transforms raw sensor measurement to a color image, is composed of a sequence of processing modules. Traditionally, the ISP pipeline is manually tuned by experts for human perception. The resulting handcrafted ISP configuration does not necessarily benefit the downstream high-level vision tasks. To mitigate these problems, this paper presents a simple yet effective framework based on Evolutionary Algorithm to search for a set of compact ISP configurations for high-level vision tasks. In particular, we encode ISP structure into a binary string and ISP parameters into a set of float numbers. Then we jointly optimize them with task-specific loss and ISP computation budgets (e.g., running time) through solving a nonlinear multi-objective optimization problem. By mutating the configurations of the ISP pipeline, we are able to remove redundant modules and design an ISP with both low cost and high accuracy. We validate the proposed method on extreme noisy and low-light raw images, and experimental results show that our framework can help find effective and efficient ISP configurations for both object detection and semantic segmentation tasks. We further provide a detailed analysis on the importance of different modules in the ISP configurations, which benefits the design of ISP for downstream tasks in the future.