Modeling 2D Appearance Evolution for 3D Object Categorization

Abstract

3D object categorization is a non-trivial task in computer vision encompassing many real-world applications. We pose the problem of categorizing 3D polygon meshes as learning appearance evolution from multi-view 2D images. Given a corpus of 3D polygon meshes, we first render the corresponding RGB and depth images from multiple viewpoints on a uniform sphere. Using rank pooling, we propose two methods to learn the appearance evolution of the 2D views. Firstly, we train view-invariant models based on a deep convolutional neural network (CNN) using the rendered RGB-D images and learn to rank the first fully connected layer activations and, therefore, capture the evolution of these extracted features. The parameters learned during this process are used as the 3D shape representations. In the second method, we learn the aggregation of the views from the outset by employing the ranking machine to the rendered RGB- D images directly, which produces aggregated 2D images which we term as ``3D shape images". We then learn CNN models on this novel shape representation for both RGB and depth which encode salient geometrical structure of the polygon. Experiments on the ModelNet40 and ModelNet10 datasets show that the proposed method consistently outperforms existing state-of-the-art algorithms in 3D shape recognition.