3D Shape Retrieval Research Articles

Editorial issue 31.2

This issue contains three papers. In the first paper, M. Hassaballah, Abdelraheem M. Aly, and Ahmad Abdelnaim, from South Valley University Qena, Egypt, propose for the Interactive simulations of fluids flow to obtain the stable pressure from modifications in the source term of pressure Poisson equation (PPE). The particles disorders are solved using a shifting technique with the current treatment of source term in PPE. In addition, the dummy boundary particles are used for the rigid boundary treatment. Performance of the stabilized ISPH model is tested on various numerical simulations with largely distorted free surface including liquid sloshing problems, fluid–fluid, fluid–structure interactions and dam-break over horizontal and inclined plane. To extend the applicability of the stabilized ISPH model, the postprocess including visual realism with the highly rendering scheme is coupled. In the second paper, Xianxuan Lin, Wang Xun, Jinyu Li, Bailin Yang, Kaili Zhang, Tianxiang Wei, from Zhejiang Gongshang University in Hangzhou, China, and Frederick W. B. Li, from Durham University, UK, propose a color migration framework for home scene images. It picks the coloring from a template image and transforms such coloring to a home scene image through a simple interaction. Their framework comprises three main parts. First, they carry out an interactive segmentation to divide an image into local regions and extract their corresponding colors. Second, they generate a matching color table by sampling the template image according to the color structure of the original home scene image. Finally, they transform colors from the matching color table to the target home scene image with boundary transition maintained. Experiment results demonstrated their framework can produce aesthetically attractive home scene coloring. In the last paper, Xiaoyu Cui, Ruifan Cai, Xiangjun Tang, Xiaogang Jin, from Zhejiang University, in Hangzhou, China, and Zhigang Deng, from University of Houston, Texas, United States, propose a novel shape-constrained fireworks simulation method with rich textures in a helmet-mounted display virtual environment using sketched feature lines as input. Their approach first retrieves an object from a three-dimensional (3D) model database using a sketch-based 3D shape retrieval algorithm. Then, in order to approximate models with complex structures, they introduce a novel point sampling algorithm based on Gaussian curvatures, which stores not only the positions of the selected vertices but also the texture (UV) coordinates information for texture display. In addition, they introduce a multilevel explosion process so that the fireworks can dynamically form specific, visually pleasing shapes. Through their experiments, they demonstrate that their approach can produce better results than state-of-the-art approaches.

Sketch‐based shape‐constrained fireworks simulation in head‐mounted virtual reality

AbstractIn this paper we present a novel shape‐constrained fireworks simulation method with rich textures in an HMD (Helmet Mounted Display) virtual environment using sketched feature lines as input. Our approach first retrieves an object from a three‐dimensional (3D) model database using a sketch‐based 3D shape retrieval algorithm. Then, in order to approximate models with complex structures, we introduce a novel point sampling algorithm based on Gaussian curvatures, which stores not only the positions of the selected vertices but also the texture (UV) coordinates information for texture display. In addition, we introduce a multilevel explosion process so that the fireworks can dynamically form specific, visually pleasing shapes. Through our experiments, we demonstrate that our approach can produce better results than state‐of‐the‐art approaches.

Recognition of 3D Shapes Based on 3V-DepthPano CNN

This paper proposes a convolutional neural network (CNN) with three branches based on the three-view drawing principle and depth panorama for 3D shape recognition. The three-view drawing principle provides three key views of a 3D shape. A depth panorama contains the complete 2.5D information of each view. 3V-DepthPano CNN is a CNN system with three branches designed for depth panoramas generated from the three key views. This recognition system, i.e., 3V-DepthPano CNN, applies a three-branch convolutional neural network to aggregate the 3D shape depth panorama information into a more compact 3D shape descriptor to implement the classification of 3D shapes. Furthermore, we adopt a fine-tuning technique on 3V-DepthPano CNN and extract shape features to facilitate the retrieval of 3D shapes. The proposed method implements a good tradeoff state between higher accuracy and training time. Experiments show that the proposed 3V-DepthPano CNN with 3 views obtains approximate accuracy to MVCNN with 12/80 views. But the 3V-DepthPano CNN frame takes much shorter time to obtain depth panoramas and train the network than MVCNN. It is superior to all other existing advanced methods for both classification and shape retrieval.

Triplanar convolution with shared 2D kernels for 3D classification and shape retrieval

Increasing the depth of Convolutional Neural Networks (CNNs) has been recognized to provide better generalization performance. However, in the case of 3D CNNs, stacking layers increases the number of learnable parameters linearly, making it more prone to learn redundant features. In this paper, we propose a novel 3D CNN structure that learns shared 2D triplanar features viewed from the three orthogonal planes, which we term S3PNet. Due to the reduced dimension of the convolutions, the proposed S3PNet is able to learn 3D representations with substantially fewer learnable parameters. Experimental evaluations show that the combination of 2D representations on the different orthogonal views learned through the S3PNet is sufficient and effective for 3D representation, with the results outperforming current methods based on fully 3D CNNs. We support this with extensive evaluations on widely used 3D data sources in computer vision: CAD models, LiDAR point clouds, RGB-D images, and 3D Computed Tomography scans. Experiments further demonstrate that S3PNet has better generalization capability for smaller training sets, and learns more of kernels with less redundancy compared to kernels learned from 3D CNNs.

MPAN: Multi-Part Attention Network for Point Cloud Based 3D Shape Retrieval

3D shape retrieval is an important researching field due to its wide applications in computer vision and multimedia fields. With the development of deep learning technology, great progress has been made in recent years and lots of methods have achieved promising 3D shape retrieval results. Due to the effective description of point cloud data on structural information for 3D shapes, lots of methods based on point cloud data format are proposed for better shape representation. However, most of them focus on extracting a global descrisptor from the whole 3D shape while the local features and detailed structural information are ignored, which negatively affect the effectiveness of shape descriptors. In addition, these methods also ignore the correlations among different parts of point clouds, which may introduce redundant information to the final shape descriptors. In order to address these issues, we propose a Multi-part attention network (MPAN) for 3D model retrieval based on point cloud. Firstly, we segment a 3D shape into multiple parts by employing a pre-trained PointNet++ segmentation model. After extracting the local features from them, we introduce a novel self-attention mechanism to explore the correlations between different parts. Meanwhile, by considering the structural relevance of them, the redundancy for representing 3D shapes is removed while the effective information is utilized. Finally, informative and discriminative shape descriptors, considering both local features and spatial correlations, are generated for 3D shape retrieval task. To validate the effectiveness of our method, we conduct several experiments on the public 3D shape benchmark, ShapeNetPart dataset. Experimental results and comparisons with state-of-the-art methods demonstrate the superiority of our proposed method.

Second-order Spectral Transform Block for 3D Shape Classification and Retrieval.

In this paper, we propose a novel network block, dubbed as second-order spectral transform block, for 3D shape retrieval and classification. This network block generalizes the second-order pooling to 3D surface by designing a learnable non-linear transform on the spectrum of the pooled descriptor. The proposed block consists of following two components. First, the second-order average (SO-Avr) and max-pooling (SOMax) operations are designed on 3D surface to aggregate local descriptors, which are shown to be more discriminative than the popular average-pooling or max-pooling. Second, a learnable spectral transform parameterized by mixture of power function is proposed to perform non-linear feature mapping in the space of pooled descriptors, i.e., manifold of symmetric positive definite matrix for SO-Avr, and space of symmetric matrix for SOMax. The proposed block can be plugged into existing network architectures to aggregate local shape descriptors for boosting their performance. We apply it to a shallow network for nonrigid 3D shape analysis and to existing networks for rigid shape analysis, where it improves the first-tier retrieval accuracy by 7.2% on SHREC'14 Real dataset and achieves state-of-the-art classification accuracy on ModelNet40. As an extension, we apply our block to 2D image classification, showing its superiority compared with traditional second-order pooling methods. We also provide theoretical and experimental analysis on stability of the proposed second-order spectral transform block.

Hamming Embedding Sensitivity Guided Fusion Network for 3D Shape Representation.

Three-dimensional multi-modal data are used to represent 3D objects in the real world in different ways. Features separately extracted from multimodality data are often poorly correlated. Recent solutions leveraging the attention mechanism to learn a joint-network for the fusion of multimodality features have weak generalization capability. In this paper, we propose a hamming embedding sensitivity network to address the problem of effectively fusing multimodality features. The proposed network called HamNet is the first end-to-end framework with the capacity to theoretically integrate data from all modalities with a unified architecture for 3D shape representation, which can be used for 3D shape retrieval and recognition. HamNet uses the feature concealment module to achieve effective deep feature fusion. The basic idea of the concealment module is to re-weight the features from each modality at an early stage with the hamming embedding of these modalities. The hamming embedding also provides an effective solution for fast retrieval tasks on a large scale dataset. We have evaluated the proposed method on the large-scale ModelNet40 dataset for the tasks of 3D shape classification, single modality and cross-modality retrieval. Comprehensive experiments and comparisons with state-of-the-art methods demonstrate that the proposed approach can achieve superior performance.

An ELM-Embedded Deep Learning Based Intelligent Recognition System for Computer Numeric Control Machine Tools

In modern manufacturing industry featured with automation and flexibility, the intelligent tool management for Computer Numeric Control (CNC) machine plays an essential role in manufacturing automation. The automatic tool recognition in terms of geometric shapes, materials and usage functions could facilitate the seamless integration with downstream process planning and scheduling processes. In this paper, a intelligent tool recognition system is proposed with a novel hybrid framework of multi-channel deep learning network with non-iterative and fast feedforward neural network to meet high efficiency and accuracy requirement in intelligent manufacturing. The combination of the fine-tuning Convolutional Neural Networks (CNNs) with the random parameter assignment mechanism of Extreme Learning Machines (ELMs) reach a balance in accurate feature extraction and fast recognition. In the proposed hybrid framework, features extracted from efficient CNNs are aggregated into robust ELM auto-encoders (ELM-AEs) to generate the compact but rich feature information, which are then feed to the subsequent single layer ELM network for tool recognition. The performance of proposed framework is verified on several standardized 3D shape retrieval and classification dataset, as well as on a self-constructed multi-view 3D data represented tool library database. Numerical experiments reveal a promising application perspective of proposed intelligent recognition system on manufacturing automation.

Spherical Harmonic Energy Over Gaussian Sphere for Incomplete 3D Shape Retrieval

Spherical harmonic (SH) has shown excellent advantages in terms of accuracy and efficiency in the retrieval of complete three-dimensional shapes. However, since the spherical function directly takes the shape centroid as the global reference point, the SH features depend heavily on the central position. In this context, the features become no longer reliable in the query of incomplete shapes that may cause erratic centroid. In this work, we propose a novel shape descriptor, namely spherical harmonic energy over the Gaussian sphere (SHE-GS), especially for the incomplete shape retrieval. Firstly, all unit normal vectors on the shape surface are mapped to points on a Gaussian sphere, which has the constant center. Secondly, kernel density estimation is used to establish a Gaussian Sphere Model (GSM) to describe the density change of these mapping points. Finally, the shape descriptor is generated by applying an SH transformation on the model. According to the way of GSM being regarded as a surface model or a volume model, we have separately designed two specific algorithm implementations. Experimental results, on two engineering shape sets containing artificially defective shapes, indicate that the proposed method outperforms other traditional methods also defined in the sphere space for incomplete shape retrieval. The superiority is verified for both similar objects in the same category or the single specific object of query.

Learning Discriminative and Generative Shape Embeddings for Three-Dimensional Shape Retrieval

As an important solution for 3D shape retrieval, a multi-view shape descriptor has achieved impressive performance. One crucial part of view-based shape descriptors is to interpret 3D structures through various 2D observations. Most existing methods like MVCNN believe that a strong classification model trained with deep learning, can often provide an efficient shape embedding for 3D shape retrieval. However, these methods pay much attention to discriminative models and none of them necessarily incorporate the underlying 3D properties of the objects from 2D images. In this paper, we present a novel encoder-decoder recurrent feature aggregation network (ERFA-Net) to address this problem. Aiming at emphasizing the 3D properties of 3D shapes in the fusion of multiple view features, 3D properties prediction tasks are introduced into the 3D shape retrieval. Specifically, an image sequence of the shape is recurrently aggregated into a discriminative shape embedding based on LSTM network, and then this latent shape embedding is trained to predict the original voxel grids and estimate images of unseen viewpoints. This generation task gives an effective supervision which makes the network exploit 3D properties of shapes through various 2D images. Our method achieves the state-of-the-art performance for 3D shape retrieval, on two large-scale 3D shape datasets, ModelNet and ShapeNetCore55. Extensive experiments show that the proposed 3D representation performs robust discrimination against view occlusion, and strong generation ability for various 3D shape tasks.

A repeatable and robust local reference frame for 3D surface matching

Local reference frames (LRFs) have been widely used for 3D local surface description. In this work, we propose a repeatable LRF with strong robustness to different nuisances. Different from existing LRF methods, the proposed LRF uses a part of neighboring points within the support region to calculate the z-axis, and performs an effective feature transformation on the neighboring points to define the x-axis. Specifically, feature transformation is applied to the data on a projection plane based on three point distribution characteristics via weighted strategies. These characteristics include the z-height, the distance to the center and the average length to 1-ring neighbors, covariance analysis is then applied to the transformed points to obtain the eigenvector with the largest eigenvalue, which points towards the maximum variance direction. Using a sign disambiguation technique, the modified eigenvector is used to define the final x-axis. Furthermore, a scale strategy is proposed to improve the robustness of the LRF with respect to mesh decimation. The proposed LRF was rigorously tested on six public benchmark datasets consisting of three different application contexts, i.e., 3D shape retrieval, 3D object recognition and registration. Experiments show that our method achieves significantly higher repeatability and stronger robustness than the state-of-the-arts under Gaussian noise, shot noise and mesh resolution variation. Finally, the descriptor matching results on four typical datasets further demonstrate the effectiveness of our LRF.

Learning SO(3) Equivariant Representations with Spherical CNNs

We address the problem of 3D rotation equivariance in convolutional neural networks. 3D rotations have been a challenging nuisance in 3D classification tasks requiring higher capacity and extended data augmentation in order to tackle it. We model 3D data with multi-valued spherical functions and we propose a novel spherical convolutional network that implements exact convolutions on the sphere by realizing them in the spherical harmonic domain. Resulting filters have local symmetry and are localized by enforcing smooth spectra. We apply a novel pooling on the spectral domain and our operations are independent of the underlying spherical resolution throughout the network. We show that networks with much lower capacity and without requiring data augmentation can exhibit performance comparable to the state of the art in standard 3D shape retrieval and classification benchmarks.

Training deep convolutional neural networks to acquire the best view of a 3D shape

In a 3D shape retrieval system, when attempting to select the best view from many view images, the ability to project a 3D shape into related view images from multiple viewpoints is important. Furthermore, learning the best view from benchmark sketch datasets is one of the best approaches to acquire the best view of a 3D shape. In this paper, we propose a learning framework based on deep neural networks to obtain the best shape views. We apply transfer learning to obtain features, i.e., we use two Alex convolutional neural networks (CNNs) for feature extraction: one for the view images and the other for the sketches. Specifically, the connections to learn an automatic best-view selector for different types of 3D shapes are obtained through the proposed learning framework. We perform training on the Shape Retrieval Contest’s 2014 Sketch Track Benchmark (SHREC’14) to capture the related rules. Finally, we report experiments to demonstrate the feasibility of our approach. In addition, to better evaluate our proposed framework and show its superiority, we apply our proposed approach to a sketch-based model retrieval task, where it outperforms other state-of-the-art methods.

A Unified Feature Representation and Learning Framework for 3D Shape

In conventional 3D shape retrieval and classification, they differentiate each other in their final stages. We propose a unified feature representation and learning framework for the instance-based shape retrieval and classification. Firstly, we render every 3D model in several directions and use the produced view-sets to represent the 3D models. In this way, both tasks can be tackled by measuring the distances between rendered views of 3D models. Secondly, we construct the viewsets as Symmetric positive definite matrices (SPDMs), which are points on a Riemannian manifold. Thus, the shape retrieval and classification tasks are reduced to a problem of measuring the distances between projected views and SPDMs. To solve this heterogeneous problem, we map them to a Hilbert space using a method of point-to-set matching. In this Hilbert space, the distances are surprisingly easy to calculate. Finally, we use a robust nearest-neighbor approach to unify the instancebased shape retrieval and classification. Our framework combines the state-of-the-art deep learning approaches with traditional mathematical optimization method, makes full use of both advantages, which is much more flexible than pure deep learning methods. Experimental results show the efficiency of our approach.

Web3D learning framework for 3D shape retrieval based on hybrid convolutional neural networks

With the rapid development of Web3D technologies, sketch-based model retrieval has become an increasingly important challenge, while the application of Virtual Reality and 3D technologies has made shape retrieval of furniture over a web browser feasible. In this paper, we propose a learning framework for shape retrieval based on two Siamese VGG-16 Convolutional Neural Networks (CNNs), and a CNN-based hybrid learning algorithm to select the best view for a shape. In this algorithm, the AlexNet and VGG-16 CNN architectures are used to perform classification tasks and to extract features, respectively. In addition, a feature fusion method is used to measure the similarity relation of the output features from the two Siamese networks. The proposed framework can provide new alternatives for furniture retrieval in the Web3D environment. The primary innovation is in the employment of deep learning methods to solve the challenge of obtaining the best view of 3D furniture, and to address cross-domain feature learning problems. We conduct an experiment to verify the feasibility of the framework and the results show our approach to be superior in comparison to many mainstream state-of-the-art approaches.

Deep point-to-subspace metric learning for sketch-based 3D shape retrieval

One key issue in managing a large scale 3D shape dataset is to identify an effective way to retrieve a shape-of-interest. The sketch-based query, which enjoys the flexibility in representing the user’s intention, has received growing interests in recent years due to the popularization of the touchscreen technology. Essentially, the sketch depicts an abstraction of a shape in a certain view while the shape contains the full 3D information. Matching between them is a cross-modality retrieval problem, and the state-of-the-art solution is to project the sketch and the 3D shape into a common space with which the cross-modality similarity can be calculated by the feature similarity/distance within. However, for a given query, only part of the viewpoints of the 3D shape is representative. Thus, blindly projecting a 3D shape into a feature vector without considering what is the query will inevitably bring query-unrepresentative information. To handle this issue, in this work we propose a Deep Point-to-Subspace Metric Learning (DPSML) framework to project a sketch into a feature vector and a 3D shape into a subspace spanned by a few selected basis feature vectors. The similarity between them is defined as the distance between the query feature vector and its closest point in the subspace by solving an optimization problem on the fly. Note that, the closest point is query-adaptive and can reflect the viewpoint information that is representative to the given query. To efficiently learn such a deep model, we formulate it as a classification problem with a special classifier design. To reduce the redundancy of 3D shapes, we also introduce a Representative-View Selection (RVS) module to select the most representative views of a 3D shape. By conducting extensive experiments on various datasets, we show that the proposed method can achieve superior performance over its competitive baseline methods and attain the state-of-the-art performance.

MLVCNN: Multi-Loop-View Convolutional Neural Network for 3D Shape Retrieval

3D shape retrieval has attracted much attention and become a hot topic in computer vision field recently.With the development of deep learning, 3D shape retrieval has also made great progress and many view-based methods have been introduced in recent years. However, how to represent 3D shapes better is still a challenging problem. At the same time, the intrinsic hierarchical associations among views still have not been well utilized. In order to tackle these problems, in this paper, we propose a multi-loop-view convolutional neural network (MLVCNN) framework for 3D shape retrieval. In this method, multiple groups of views are extracted from different loop directions first. Given these multiple loop views, the proposed MLVCNN framework introduces a hierarchical view-loop-shape architecture, i.e., the view level, the loop level, and the shape level, to conduct 3D shape representation from different scales. In the view-level, a convolutional neural network is first trained to extract view features. Then, the proposed Loop Normalization and LSTM are utilized for each loop of view to generate the loop-level features, which considering the intrinsic associations of the different views in the same loop. Finally, all the loop-level descriptors are combined into a shape-level descriptor for 3D shape representation, which is used for 3D shape retrieval. Our proposed method has been evaluated on the public 3D shape benchmark, i.e., ModelNet40. Experiments and comparisons with the state-of-the-art methods show that the proposed MLVCNN method can achieve significant performance improvement on 3D shape retrieval tasks. Our MLVCNN outperforms the state-of-the-art methods by the mAP of 4.84% in 3D shape retrieval task. We have also evaluated the performance of the proposed method on the 3D shape classification task where MLVCNN also achieves superior performance compared with recent methods.

Angular Triplet-Center Loss for Multi-View 3D Shape Retrieval

How to obtain the desirable representation of a 3D shape, which is discriminative across categories and polymerized within classes, is a significant challenge in 3D shape retrieval. Most existing 3D shape retrieval methods focus on capturing strong discriminative shape representation with softmax loss for the classification task, while the shape feature learning with metric loss is neglected for 3D shape retrieval. In this paper, we address this problem based on the intuition that the cosine distance of shape embeddings should be close enough within the same class and far away across categories. Since most of 3D shape retrieval tasks use cosine distance of shape features for measuring shape similarity, we propose a novel metric loss named angular triplet-center loss, which directly optimizes the cosine distances between the features. It inherits the triplet-center loss property to achieve larger inter-class distance and smaller intra-class distance simultaneously. Unlike previous metric loss utilized in 3D shape retrieval methods, where Euclidean distance is adopted and the margin design is difficult, the proposed method is more convenient to train feature embeddings and more suitable for 3D shape retrieval. Moreover, the angle margin is adopted to replace the cosine margin in order to provide more explicit discriminative constraints on an embedding space. Extensive experimental results on two popular 3D object retrieval benchmarks, ModelNet40 and ShapeNetCore 55, demonstrate the effectiveness of our proposed loss, and our method has achieved state-ofthe-art results on various 3D shape datasets.

Multi-feature distance metric learning for non-rigid 3D shape retrieval

In the past decades, feature-learning-based 3D shape retrieval approaches have been received widespread attention in the computer graphic community. These approaches usually explored the hand-crafted distance metric or conventional distance metric learning methods to compute the similarity of the single feature. The single feature always contains onefold geometric information, which cannot characterize the 3D shapes well. Therefore, the multiple features should be used for the retrieval task to overcome the limitation of single feature and further improve the performance. However, most conventional distance metric learning methods fail to integrate the complementary information from multiple features to construct the distance metric. To address these issue, a novel multi-feature distance metric learning method for non-rigid 3D shape retrieval is presented in this study, which can make full use of the complimentary geometric information from multiple shape features by utilizing the KL-divergences. Minimizing KL-divergence between different metric of features and a common metric is a consistency constraints, which can lead the consistency shared latent feature space of the multiple features. We apply the proposed method to 3D model retrieval, and test our method on well known benchmark database. The results show that our method substantially outperforms the state-of-the-art non-rigid 3D shape retrieval methods.

A dimensional reduction guiding deep learning architecture for 3D shape retrieval

The state-of-the-art shape descriptors are usually lengthy for gaining high retrieval precision. With the rapidly growing number of 3-dimensional models, the retrieval speed becomes a prominent problem in shape retrieval. In this paper, by exploiting the capabilities of the dimensionality reduction methods and the deep convolutional residual network (ResNet), we developed a method for extracting short shape descriptors (with just 2 real numbers, named 2-descriptors) from lengthy descriptors, while keeping or even improving the retrieval precision of the original lengthy descriptors. Specifically, an attraction and repulsion model is devised to strengthen the direct dimensionality reduction results. In this way, the dimensionality reduction results turn into desirable labels for the ResNet. Moreover, to extract the 2-descriptors using ResNet, we transformed it as a classification problem. For this purpose, the range of each component of the dimensionality reduction results (including two components in total) is uniformly divided into n intervals corresponding to n classes. Experiments on 3D shape retrieval show that our method not only accelerates the retrieval speed greatly but also improves the retrieval precisions of the original shape descriptors.