Disentangled and controllable sketch creation based on disentangling the structure and color enhancement

Abstract

Existing sketch-based image processing methods include sketch recognition, sketch synthesis and sketch-based image retrieval. For sketch creation, a meaningful task is proposed namely disentangled and controllable sketch creation (DCSC) based on disentangling the structure and color enhancement. Specifically, as the first subtask, sketch structure enhancement (SSE) is used to enhance a non-professional sketch (NPS) and obtain a professional sketch (PS), which is a process denoted as NPS2PS. A data set named SketchMan is first provided, consisting of NPSs and PSs with various postures in different scenes. SSE is trained as a conditional image-to-image translation problem, and there are three models: direct sketch-to-sketch (SS), grayscale guided SS and contour guided SS. Multiple IOU metrics are proposed based on Corner Point Map (CPM), Straight Line Map (SLM) and Segmented Area Map (SAM). As the second subtask, sketch color enhancement (SCE) is trained as a two-stage framework containing a topology enhancement network (TE-Net) that maps a sketch to the corresponding grayscale domain and a color injection network (CI-Net) that injects the global color feature to the AdaIN residual blocks to perform adaptive sketch colorization. The TE-Net and CI-Net disentangle the topological and color features to perform more controllable and diverse SCE results. Experimental results demonstrate that our proposed methods are effective to address the challenging and meaningful DCSC task compared with other state-of-the-art methods.