Deep-learning based single-shot 3D reconstruction with simulated color-crosstalk and randomized extrinsics

Abstract

In recent years, many single-frame 3D reconstruction schemes based on fringe projection profilometry (FPP) have been proposed. However, most single-frame reconstruction schemes still face the following three issues: (1) obtaining large datasets is very time-consuming, (2) focusing only on achieving single-frame reconstruction for white objects, and (3) requiring fixing the camera-projector positional relationship, increasing operational difficulty. By building a virtual FPP simulation system, our method can quickly render the required datasets, avoiding cumbersome manual operations. When rendering the datasets, we simulate adverse factors such as color channel crosstalk, system extrinsic parameter variations, and object surface colors to guide the training of the neural network. Ultimately, from a single three-frequency color image, the corresponding three-frequency three-step phase-shift images are predicted, achieving single-frame 3D reconstruction of colored objects and allowing some variation in system extrinsic parameters. Real-world experiments demonstrate that the network trained with the diverse data generated by our virtual system has good accuracy, providing valuable guidance for the practical application of deep learning methods.