HiStyle: Reinventing historic portraits via 3D generative model

Abstract

Recreating historical portraits with accuracy and artistic diversity has always been a challenge in the field of computer vision. To ensure faithful reinvention of portrait images, it is essential to not only restore colors and reconstruct 3D geometry but also incorporate various artistic styles. Although significant progress has been made in individual tasks, existing methods often struggle with a trade-off between low-quality yet accurate restoration, limiting their ability to meet all criteria within a unified model. To address these challenges, we propose HiStyle, a generative model that simultaneously supports 2D to 3D reconstruction, grayscale to RGB conversion, and photo-to-stylized image transformation. HiStyle first introduces a GAN inversion technique, restoring the lost color information of input historic portraits while elevating 2D images to 3D representation. Additionally, we integrate the powerful CLIP model into 3D-aware GANs to achieve zero-shot text-driven style transfer. To further enhance the range of styles, we leverage the latent diffusion model to synthesize multiple 2D style extensions of the colorized images. Experiment results demonstrate improved quality and diversity of generated images. Our HiStyle reveals the potential of 3D-aware GANs in preserving cultural heritage.