Point-StyleGAN: Multi-scale point cloud synthesis with style modulation

Abstract

A point cloud is a set of discrete surface samples. As the simplest 3D representation, it is widely used in 3D reconstruction and perception. Yet developing a generative model for point clouds remains challenging due to the sparsity and irregularity of points. Drawn by StyleGAN, the forefront image generation model, this paper presents Point-StyleGAN, a generator adapted from StyleGAN2 architecture for point cloud synthesis. Specifically, we replace all the 2D convolutions with 1D ones and introduce a series of multi-resolution discriminators to overcome the under-constrained issue caused by the sparsity of points. We further add a metric learning-based loss to improve generation diversity. Besides the generation task, we show several applications based on GAN inversion, among which an inversion encoder Point-pSp is designed and applied to point cloud reconstruction, completion, and interpolation. To our best knowledge, Point-pSp is the first inversion encoder for point cloud embedding in the latent space of GANs. The comparisons to prior work and the applications of GAN inversion demonstrate the advantages of our method. We believe the potential brought by the Point-StyleGAN architecture would further inspire massive follow-up works.