Painting patches: Reducing flicker in painterly re-rendering of video

Abstract

This paper presents a novel method for re-rendering video in a stroke-based painterly style. Previous methods typically place and adjust strokes on a frame by frame basis, guided by an analysis of motion vectors. Our method constructs painting patches which last for multiple frames, and paints them just once, compositing them after placing and clipping each one in each output frame. Painting patches are constructed by clustering pixels with similar motions, representing moving objects. This is done using a multi-frame window, to take account of objects which are present in consecutive frames, and which may occur a few frames apart with occlusion. The appearance of a given cluster across a sequence of frames is warped to a common reference to produce the painting patch; a global optimization of the warp is used to minimize distortion in the painting strokes. This approach outperforms prior algorithms in problem areas of the image, where flickering typically occurs, while producing comparable results elsewhere. In particular, stable strokes are produced at occlusion boundaries where objects emerge, and at image borders exposed by camera panning. A further advantage is consistent rendering of regions before and after brief occlusion, enhancing temporal stability of the output of discontiguous frames.