Abstract
In this article we introduce a new self-supervised, semi-parametric approach for synthesizing novel views of a vehicle starting from a single monocular image. Differently from parametric (i.e., entirely learning-based) methods, we show how a-priori geometric knowledge about the object and the 3D world can be successfully integrated into a deep learning based image generation framework. As this geometric component is not learnt, we call our approach semi-parametric. In particular, we exploit man-made object symmetry and piece-wise planarity to integrate rich a-priori visual information into the novel viewpoint synthesis process. An Image Completion Network (ICN) is then trained to generate a realistic image starting from this geometric guidance. This careful blend between parametric and non-parametric components allows us to i) operate in a real-world scenario, ii) preserve high-frequency visual information such as textures, iii) handle truly arbitrary 3D roto-translations of the input, and iv) perform shape transfer to completely different 3D models. Eventually, we show that our approach can be easily complemented with synthetic data and extended to other rigid objects with completely different topology, even in presence of concave structures and holes (e.g., chairs). A comprehensive experimental analysis against state-of-the-art competitors shows the efficacy of our method both from a quantitative and a perceptive point of view. Supplementary material, animated results, code, and data are available at: https://github.com/ndrplz/semiparametric.
Highlights
H OW would you see an object from another point of view? Given a single view of an object in the world, predicting how it would look like from arbitrarily different viewpoints is definitely non-trivial for both humans and machines
Powerful parametric deep learning models [27], [15] made it possible to frame the generation of novel viewpoints as a conditioned image synthesis problem
We propose an original formulation of the problem of object novel viewpoint synthesis in a semi-parametric setting
Summary
H OW would you see an object from another point of view? Given a single view of an object in the world, predicting how it would look like from arbitrarily different viewpoints is definitely non-trivial for both humans and machines. Powerful parametric (i.e. entirely learning-based) deep learning models [27], [15] made it possible to frame the generation of novel viewpoints as a conditioned image synthesis problem This is an holistic approach that under-exploits the fact that man-made objects 3D models are roughly distributed according to few prototypes Vast amount of data are required for the network to generalize to arbitrary transformations (i.e. a sufficient number of images for every possible viewpoint). This constrains many methods to be trained solely on synthetic data
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have