Multi-modal fusion network guided by prior knowledge for 3D CAD model recognition

Abstract

Currently, the 3D CAD model has found extensive use in various applications. The more accurate recognition and more reliable reuse of 3D computer-aided design (CAD) models could significantly save labor costs and time in the production and design of 3D products. However, existing 3D model recognition methods are constrained by a single representation form that lacks supplementary information from multiple sources, and there is a lack of corresponding guidance for relationships between different data. In this paper, we focus on multi-modal information for 3D model representation and propose a Multi-modal Fusion Network (MMFN) guided by prior knowledge for 3D CAD model recognition. In particular, we consider prior knowledge about the class distribution of 3D models: The designed labeling information is utilized as the prior knowledge to guide the multi-modal information fusion via the cross-attention structure. Then, the contrastive learning method is utilized in the optimization step, further increasing the aggregation of similar samples and contributing to enhancing the discriminative capability of features. Finally, we conduct extensive experiments on the public datasets, ModelNet, ShapeNet, and a challenging industrial 3D CAD dataset built by ourselves. Compared to state-of-the-art approaches, our MMFN provides competitive results. The source code has been published on Github: https://github.com/WhiteTJU/MMFN.