Free-hand sketches for 3D model retrieval using cascaded LSDA

Abstract

This study proposes a novel cascaded 3D model retrieval framework for automatically and accurately fitting 3D models to user freehand 2D sketches (or object contours). The matching result provides the 3D geometry hypotheses of the 2D objects, and thus it is an important and fundamental part of several computer vision tasks, such as spatial layout estimation and scene understanding. However, existing matching approaches are still restricted by the corresponding point/line matching between the 2D image and the target 3D model. The limitation may make existing matching methods not online accessible in practice, and degrade their applicability. An alternative scenario is proposed to address this problem, in this paper. We use a retrieval system to collect a 3D-2D database offline, in which the 3D models are decomposed into 2D projected contours with respective to a limited number of different viewpoints. Our approach aims to enhance matching results by leveraging the extra database. Specifically, it optimizes a transformation from 3D objects to 2D contours, by which the target 3D object pose and categorization, to be recognized can be concisely estimated by entries in such database. More importantly, we demonstrate how to use a set of training images with limited pose variation to handle test images taken under uncontrolled view.