Gradual adaption with memory mechanism for image-based 3D model retrieval

Abstract

With the development of 3D modeling technology and its wide application in different fields, the number of 3D models increases rapidly, making 3D model retrieval a hot topic in current research. Compared with other 3D model retrieval methods, 2D image-based unsupervised 3D model retrieval takes the 2D images which have rich labels and are easy to obtain as the queries, and also takes into account the difficulties of labeling 3D models. 2D image-based unsupervised 3D model retrieval is a retrieval task involving cross-domain adaptation problem, which main challenge is the excessive domain gap. In this paper, we propose a cross-domain 3D model retrieval method of memory mechanism based on disentangled feature learning. The disentangled feature learning enables to disentangle the twisted original features into the isolated domain-invariant features and domain-specific features, where the former is to be aligned to narrow the domain gap. On this basis, the memory mechanism selects feature vectors from class memory modules constructed by class representative features of the opposite domain for every sample, which are used to update the domain-invariant features with gradient weight. The memory mechanism can gradually improve the adaptability of the model to the very different two domains. Experiments are conducted on the public datasets MI3DOR and MI3DOR-2 to verify the feasibility and the superiority of the proposed method. Especially on MI3DOR-2 dataset, our method outperforms the current state-of-the-art methods with gains of 7.71% for the strictest retrieval metric NN. • An end-to-end unsupervised 2D image-based 3D model retrieval framework. • Transfering knowledge from labeled 2D images to unlabeled 3D models. • Domain-invariant features are disentangled from the original features. • Memory module enhances domain-invariant features by representative features. • Experiments on MI3DOR and MI3DOR-2 verified the superiority of the method.