Zero-Shot Single-View Point Cloud Reconstruction via Cross-Category Knowledge Transferring

Abstract

Single-view point cloud reconstruction aims to generate a 3D point cloud of an object given one 2D image taken from an arbitrary viewpoint. Most previous works assume that all the test categories have been present to the model during training. However, it is impossible to know all the test categories that the model will meet in advance. And we discover these methods can not deal with novel categories well. Therefore, in this paper, we investigate a more realistic and challenging setting of single-view point cloud reconstruction, zero-shot, where the model's performance on novel categories is pursued. Towards this task, we propose the Cross-Category Knowledge Transferring Network (CCKTN), which maintains a knowledge bank to mine transferable knowledge from known categories to help reconstruct novel categories. Additionally, we conduct auxiliary learning for the point cloud reconstruction model with the point cloud autoencoder via sharing the same knowledge bank. This design enables the knowledge bank to collect more fruitful 3D knowledge of point clouds. Moreover, we devise a diversity loss regularization for the knowledge vectors to guarantee their diversities, further enhancing CCKTN's performance. Comprehensive experiments conducted on ShapeNet and ModelNet datasets show CCKTN's superiority towards existing methods and demonstrate CCKTN's effectiveness for reconstructing novel category objects.