Text-Based Point Cloud Generation Adversarial Network

Abstract

Point clouds become the popular representation of 3D shapes. Meanwhile, deep generative adversarial networks (GANs) have been used to generate reasonable point clouds for specific categories, such as airplane, chair. But existing point clouds generation tasks are almost unconditional, which means the generative results cannot be controlled. In this work, we propose the text-based point cloud generative adversarial network short as TPC-GAN for generating point clouds from natural language. To this end, we first contribute a point clouds dataset with text description based on the chair category of ModelNet40. Then we learn the implicit cross-modal con-nections between texts and point clouds through category labels. Using a pretrained text encoder that had gained the mapping relations between texts and class labels to obtain text embedding of input text, which will concatenate with a noise vector and be feed into TPC-GAN. Moreover, the discriminators of TPC-GAN can be regarded as point clouds classifier. Finally, the TPC-GAN can generate point clouds that meet the input text description. To evaluate the performance of our approach, some experiments are conducted. To the best of our knowledge, our method is the first one to generate point clouds from texts.