Motion-guided semantic alignment for line art animation colorization

Abstract

Line art animation colorization aims to colorize a sequence of line art frames with a reference while ensuring semantic alignment across frames. Recent methods have utilized similarity metrics to maintain inter-frame semantic alignment. However, these approaches typically struggle with precise pixel-to-pixel motion alignment, resulting in issues like semantic inconsistencies and color blurring. To address these problems, we introduce an innovative Motion-Guided Colorization Network(MGC-Net), which incorporates line art motions into account for achieving coarse to fine semantic alignment. At the coarse level, our approach includes a Hierarchical Cost-feature Distillation (HCD) module. This module is designed to establish accurate pixel-to-pixel motion alignment between frames. It employs a motion distillation loss, which effectively transfers accurate color motion representation to line arts at varying hierarchical levels. Specifically, we utilize the cost features in inter-frames as a representation of motion, processed through our self-supervision operator. At the finer level, we propose the Correlation Refinement (CR) module. This module enhances semantic alignment by integrating semantic contextual dependencies between the color reference and the motion-based reference obtained from our motion-based confidence estimation block. CR module refines the final colorization process, improving clear color detail and semantic consistency across frames. Quantitative and qualitative experiments on animation datasets show that MGC-Net outperforms the state-of-the-art methods.