Camera ISP Modification to Enable Image De-rendering

Abstract

A camera's image signal processor (ISP) is dedicated hardware that performs a series of processing steps to render a captured raw sensor image to its final display-referred output suitable for viewing and sharing. It is often desirable to be able to revert – or de-render – the ISP-processed image back to the original raw sensor image. Undoing the ISP rendering, however, is not an easy task. This is because ISPs perform many nonlinear routines in the rendering pipeline that are difficult to invert. Moreover, modern cameras often apply scene-specific image processing, resulting in a wide range of possible ISP parameters. In this paper, we propose a modification to the ISP that allows the ISP-rendered image to be reverted back to a raw image. Our approach works by appending a fixed-sampling of the raw sensor values to all captured images. The appended raw samples comprise no more than 8 rows of pixels in the full-sized image and represent a negligible overhead given that 12–16 MP sensors typically have 3000 rows of pixels or more. The appended pixels are rendered along with the captured image to the final output. From these rendered raw samples, a reverse mapping function can be computed to undo the ISP processing. We demonstrate that this method performs almost on par with competing state-ofthe-art approaches for ISP de-rendering while offering a practical solution that is integrable to current camera ISP hardware.