Incorporating Higher-Order Cues in Image Colorization

Abstract

Colorization problem is to find the colors of all pixels X = {xn}n=1,...,|X |, given a grayscale image I with scribbles S with the desired colors. We work in the YUV color space where Y = {yn}n=1,...,|X | is the monochromatic luminance channel, which we will refer to simply as intensity, while U = {un}n=1,...,|X | and V = {vn}n=1,...,|X | are the chrominance channels, encoding the color. Our goal is to complete both the U and V channels, given Y = I. We deal with the only U channel in this paper, since the V channel can be treated in the same manner. In this paper, we propose a new multi-layer graph model and an energy formulation that can incorporate higher-order cues for reliable colorization of natural images. In contrast to most existing energy functions [3] with unary and pairwise constraints, we address the problem of imposing a high-order constraint whereby pixels constituting each region tend to have similar colors to the representative color of the region they belong to. The representative colors of the regions that are generated by unsupervised image segmentation algorithms, act as higher-order cues. Unlike previous parametric models [2], they are automatically obtained by a nonparametric learning technique that estimates them from the resulting pixel colors in a recursive fashion. We formulate this problem in terms of two quadratic energy functions of pixel and region colors, that are supplementary to each other, in our proposed multi-layer graph model and estimate them by a simple optimization technique that minimizes both functions simultaneously. Our proposed algorithm works as follows. We first design an undirected graph G = (Q,E) where the nodes Q = {X ,R} consist of two types: pixels X and regions R, generated by an unsupervised segmentation algorithm such as Mean Shift [1], and the edges E are the links between two nodes as shown in Fig. 1(a). Each pixel xn ∈ X initially has an intensity yn ∈ Y . For each region rk ∈ R, we can generate its properties ȳk as the mean intensity of the inner pixels xn ∈ rk: ȳk = 1 |rk| ∑xn∈rk yn. We then formulate both quadratic energy functions JX and JR for estimating the pixel colors U = {un}n=1,...,|X | and the region colors Ū = {ūk}k=1,...,|R|, respectively, as follows.