S[formula omitted]U-PVNet: Arbitrary-scale point cloud upsampling via Point-Voxel Network based on Siamese Self-Supervised Learning

Abstract

Recently, benefiting from the rapid development of deep learning, many research on point cloud upsampling task has been successfully proposed with remarkable performance. The quality of the upsampled point clouds is also vital for the quality of the reconstructed meshes in 3D surface reconstruction task. However, most existing methods either independently train a specific upsampling network for each scale factor or heavily rely on the paired training data as the supervision information. To address these limitations which are both inefficient and impractical for storage and computation in real applications, we present a Point-Voxel Network based on Siamese Self-supervised learning for arbitrary-Scale point cloud Upsampling (S3U-PVNet), which mainly includes a Down–Up pipeline and an Up–Down pipeline, to support self-supervised point cloud upsampling with arbitrary scale factors by a single network. The core of our network is a series of stacked point-voxel feature fusion (PVFF) modules, which can effectively and efficiently extract and fuse multi-granularity features from input point clouds. Each module includes two branches, a point-based branch that represents the sparse inputs in points to adaptively learn the spatial relationship among points, and a voxel-based branch that extract features in voxels to reduce the problem of inaccurate feature extraction caused by the irregularity and sparsity of point clouds. Furthermore, we also propose an end-to-end training objective for our siamese self-supervised network encapsulating reconstruction loss and similarity loss, which considers both global shape constraint and local geometric constraint. We achieve new state-of-the-art unsupervised upsampling results on various synthetic datasets and demonstrate the generalization of the proposed method on challenging real-world data.