3D Shape Recognition Research Articles

Beteng Cepuri as Ethnomathematics of Elementary Schools

Ethnomathematical objects can be found in artifacts in the form of historical buildings. One of the historical buildings in Kotagede, Yogyakarta, Indoensia, is the Cepuri Beteng. The purpose of this research is an exploration any mathematical concepts contained in Beteng Cepuri that can be used as a source of learning mathematics as an effort to develop ethnomathematics for learning elementary mathematics. The researchers invited the 5th graders of an elementary school in Kotagede, nearby the Cepuri Beteng, to explore and understand mathematical concepts. The results showed that there was a relationship between the construction of the fort building and the concept of building space in several parts of the Cepuri Beteng. The sections are in the form of blocks, and rectangles. Taking the students to visualize the artifacts may assist them to build 3D shape recognition. It can be described that ethnomathematics is able to provide a very good interest in learning followed by concept acquisition.

Local information fusion network for 3D shape classification and retrieval

Due to the wide applications of 3D shapes, exploring effective 3D shape recognition algorithms has attracted much research attention. Various approaches have been proposed in recent years, within which the multi-view based methods show their promising performances. Previous multi-view based methods mainly extract global features of views by employing the well-established CNN and then explore the correlations between the view-level descriptors for 3D shape representation. However, these approaches ignore the local characteristics extraction of view images. Besides, they also lack the consideration for the relationships between image regions and feature map channels, which could provide more detailed and descriptive information to improve the discrimination of shape descriptors. To address these issues, we propose a novel Local Information Fusion Network (LIFN) for local characteristics extraction and relationships exploration based on the feature maps during the convolution process. Concretely, the Region Organization Module (ROM) is introduced for feature map reorganization and region-wise feature extraction. Besides, the Region-wise Attention (RWA) and Channel-wise Attention (CWA) are designed for region-wise and channel-wise interaction exploration, respectively. Extensive experiments on the ModelNet40 database demonstrate the superiority of our proposed network against the state-of-the-art approaches.

LIMAN: Local Information-Based Multiattention Network for 3D Shape Recognition

Exploring effective 3D shape recognition algorithms attracts much research attention recently. The existing multiview-based 3D shape recognition methods usually ignore the local information exploration in each view image. In this article, we propose a novel local information-based multiattention network, which concentrates the effective information in the feature maps and captures the view-wise interactions to improve the shape representation performance. Concretely, we design the spatial-aware and channel-aware weight learning modules to quantify the contributions of each spatial region and channel for feature map updating in the view-wise descriptor extraction process. Then, the multihead attention module is introduced for multiview feature aggregation, considering the view-wise interactions to preserve the effective information. Finally, we generate the discriminative shape descriptors for 3D shape recognition task. We conduct comprehensive experiments on the ModelNet databases and achieve competitive performances compared with the state-of-the-art methods.

3DSliceLeNet: Recognizing 3D Objects Using a Slice-Representation

Convolutional Neural Networks (CNNs) have become the default paradigm for addressing classification problems, especially, but not only, in image recognition. This is mainly due to their high success rate. Although a number of approaches currently apply deep learning to the 3D shape recognition problem, they are either too slow for online use or too error-prone. To fill this gap, we propose 3DSliceLeNet, a deep learning architecture for point cloud classification. Our proposal converts the input point clouds into a two-dimensional representation by performing a slicing process and projecting the points to the principal planes, thus generating images that are used by the convolutional architecture. 3DSliceLeNet successfully achieves both high accuracy and low computational cost. A dense set of experiments has been conducted to validate our system under the ModelNet challenge, a large-scale 3D Computer Aided Design (CAD) model dataset. Our proposal achieves a success rate of 94.37% and an Area under Curve (AUC) of 0.978 on the ModelNet-10 classification task.

Learning Representation on Optimized High-Order Manifold for Visual Classification

Graph convolutional networks (GCNs) and graph neural networks (GNNs) have demonstrated convincing performance on many tasks by learning the intrinsic structure of the data. However, it is still valuable and challenging to consider the complex and complete correlations of objects, i.e., high-order manifold structures, for representation learning. In this paper, we present a novel representation learning method that utilizes the optimized high-order manifold of the data for classification tasks of nonstructural data and graph-structure data. In the method, we fully explore the complicated relationship of samples by highlighting the high-order manifold information in a hypergraph. Specifically, we incorporate high-order manifold information by graph <inline-formula><tex-math notation="LaTeX">$p$</tex-math></inline-formula>-Laplacian into a hypergraph and propose <inline-formula><tex-math notation="LaTeX">$p$</tex-math></inline-formula>-Laplacian-based hypergraph neural networks (pLapHGNN) to significantly learn hidden layer representations that encode both the high-order structure of data and the high-order manifold geometrical information. Confronting the difficulties of obtaining optimized high-order manifolds of the data, we propose an effective approximate approach by graph <inline-formula><tex-math notation="LaTeX">$p$</tex-math></inline-formula>-Laplacian representing the relationship of hyperedges in the hypergraph. Furthermore, we study the weights of hyperedges in a hypergraph with high-order manifold information. Experiments on the ModelNet40 dataset and NTU dataset demonstrate that the proposed method is more effective than the other popular methods for 3D shape recognition. Extensive experiments on other visual classification tasks and citation networks also show the superiority of our proposed method for representation learning.

MHFP: Multi-view based hierarchical fusion pooling method for 3D shape recognition

• We propose a novel MHFP architecture, which can attain better multi-view features. • We design a 3D attention module for multi-view to construct the graph in our MHFP. • The results on ModelNet40 show the superiority of our multi-view based method. 3D shape recognition has received widespread attention in the field of computer vision. Since the 3D model contains much geometric information, which is difficult to extract but important to feature learning, effective description of 3D shape is still facing great challenges. With the rapid development of deep learning, a large number of methods have been proposed. However, these approaches always focus on the learning of view features but ignore the multi-view information protection in the process of feature fusion. In this work, we propose a novel Multi-view based Hierarchical Fusion Pooling Method (MHFP) for 3D Model Recognition, which hierarchically fuses the features of multi-view into a compact descriptor. Our approach considers the correlation between views, it can powerfully remove redundant information and retain a large amount of essential information. Meanwhile, we design a 3D attention module to dig out the correlation between the views, which prepares for the graph construction. To verify the effectiveness of our MHFP, we conduct experiments on the ModelNet40 dataset and compare it with some state-of-the-art methods. The final results demonstrate the superiority of our proposed approach in 3D shape recognition.

MV-LFN: Multi-view based local information fusion network for 3D shape recognition

3D shape recognition is a challenging task due to the difficulty of representing the complex structure of 3D shapes. Recently, the view-based approaches that utilize the multiple views rendered from the shape for visual information extraction and feature aggregation to generate a global shape descriptor, achieved promising performance. However, the view-based approaches commonly ignore the exploration and utilization of local information in the multiple views, which influences the effectiveness of generated features. In this paper, we design a novel Multi-view based Local Information Fusion Network (MV-LFN) for the 3D shape recognition task. The local correlation attention mechanism (LCAM) is introduced to exploit the local correlations in the feature maps for generating a more effective view descriptor. Then, we hierarchically aggregate the multi-view feature maps to generate a shape super matrix (SSM). The local information is effectively extracted and maintained during the multi-view aggregation process, and the discrimination of shape descriptors is significantly improved. We conduct comparative experiments on the ModelNet and ShapeNetCore55 databases. The experimental performances effectively validate the superiority of MV-LFN.

PGNet: Progressive Feature Guide Learning Network for Three-dimensional Shape Recognition

Three-dimensional (3D) shape recognition is a popular topic and has potential application value in the field of computer vision. With the recent proliferation of deep learning, various deep learning models have achieved state-of-the-art performance. Among them, multiview-based 3D shape representation has received increased attention in recent years, and related approaches have shown significant improvement in 3D shape recognition. However, these methods focus on feature learning based on the design of the network and ignore the correlation among views. In this article, we propose a novel progressive feature guide learning network (PGNet) that focuses on the correlation among multiple views and integrates multiple modalities for 3D shape recognition. In particular, we propose two information fusion schemes from visual and feature aspects. The visual fusion scheme focuses on the view level and employs the soft-attention model to define the weights of views for visual information fusion. The feature fusion scheme focuses on the feature dimension information and employs the quantified feature as the mask to further optimize the feature. These two schemes jointly construct a PGNet for 3D shape representation. The classic ModelNet40 and ShapeNetCore55 datasets are applied to demonstrate the performance of our approach. The corresponding experiment also demonstrates the superiority of our approach.

Learning representative viewpoints in 3D shape recognition

Learning representative viewpoints in 3D shape recognition

PVLNet: Parameterized-View-Learning neural network for 3D shape recognition

3D shape recognition has drawn much attention in recent years. Despite the amazing progress on view-based 3D feature description, previous multi-view based methods suffer from a burden in computation efficiency compared with point cloud based methods. To overcome the limitation, we propose a novel light-weight multi-view based network built on parameterized-view-learning mechanism, PVLNet, which can achieve the state-of-the-art performance with only 1/10 FLOPs compared with previous multi-view based methods. Guided by the parameterized-view-learning mechanism, the views are directly built as parameters of PVLNet which can be automatically optimized by gradient descent. A simplified differentiable depth map generator is used to ensure the gradient propagation when generating depth images from view parameters. Then multi-view features extracted by CNNs are aggregated by global max-pooling. Our experimental results on ModelNet40 and ScanObjectNN demonstrate the superior performance of the proposed method. The visualization of the networks attention further interprets the efficiency of our PVLNet.

Unravelling Spatial Privacy Risks of Mobile Mixed Reality Data

Previously, 3D data---particularly, spatial data---have primarily been utilized in the field of geo-spatial analyses, or robot navigation (e.g. self-automated cars) as 3D representations of geographical or terrain data (usually extracted from lidar). Now, with the increasing user adoption of augmented, mixed, and virtual reality (AR/MR/VR; we collectively refer to as MR) technology on user mobile devices, spatial data has become more ubiquitous. However, this ubiquity also opens up a new threat vector for adversaries: aside from the traditional forms of mobile media such as images and video, spatial data poses additional and, potentially, latent risks to users of AR/MR/VR. Thus, in this work, we analyse MR spatial data using various spatial complexity metrics---including a cosine similarity-based, and a Euclidean distance-based metric---as heuristic or empirical measures that can signify the inference risk a captured space has. To demonstrate the risk, we utilise 3D shape recognition and classification algorithms for spatial inference attacks over various 3D spatial data captured using mobile MR platforms: i.e. Microsoft HoloLens, and Android with Google ARCore. Our experimental evaluation and investigation shows that the cosine similarity-based metric is a good spatial complexity measure of captured 3D spatial maps and can be utilised as an indicator of spatial inference risk.

Multi-Scale Representation Learning on Hypergraph for 3D Shape Retrieval and Recognition.

Effective 3D shape retrieval and recognition are challenging but important tasks in computer vision research field, which have attracted much attention in recent decades. Although recent progress has shown significant improvement of deep learning methods on 3D shape retrieval and recognition performance, it is still under investigated of how to jointly learn an optimal representation of 3D shapes considering their relationships. To tackle this issue, we propose a multi-scale representation learning method on hypergraph for 3D shape retrieval and recognition, called multi-scale hypergraph neural network (MHGNN). In this method, the correlation among 3D shapes is formulated in a hypergraph and a hypergraph convolution process is conducted to learn the representations. Here, multiple representations can be obtained through different convolution layers, leading to multi-scale representations of 3D shapes. A fusion module is then introduced to combine these representations for 3D shape retrieval and recognition. The main advantages of our method lie in 1) the high-order correlation among 3D shapes can be investigated in the framework and 2) the joint multi-scale representation can be more robust for comparison. Comparisons with state-of-the-art methods on the public ModelNet40 dataset demonstrate remarkable performance improvement of our proposed method on the 3D shape retrieval task. Meanwhile, experiments on recognition tasks also show better results of our proposed method, which indicate the superiority of our method on learning better representation for retrieval and recognition.

DAN: Deep-Attention Network for 3D Shape Recognition.

Due to the wide applications in a rapidly increasing number of different fields, 3D shape recognition has become a hot topic in the computer vision field. Many approaches have been proposed in recent years. However, there remain huge challenges in two aspects: exploring the effective representation of 3D shapes and reducing the redundant complexity of 3D shapes. In this paper, we propose a novel deep-attention network (DAN) for 3D shape representation based on multiview information. More specifically, we introduce the attention mechanism to construct a deep multiattention network that has advantages in two aspects: 1) information selection, in which DAN utilizes the self-attention mechanism to update the feature vector of each view, effectively reducing the redundant information, and 2) information fusion, in which DAN applies attention mechanism that can save more effective information by considering the correlations among views. Meanwhile, deep network structure can fully consider the correlations to continuously fuse effective information. To validate the effectiveness of our proposed method, we conduct experiments on the public 3D shape datasets: ModelNet40, ModelNet10, and ShapeNetCore55. Experimental results and comparison with state-of-the-art methods demonstrate the superiority of our proposed method. Code is released on https://github.com/RiDang/DANN.

PVRAR: Point-View Relation Neural Network Embedded with Both Attention Mechanism and Radon Transform for 3D Shape Recognition

Owing to the favorable performance of deep neural networks for 3D shape recognition, an increasing number of researchers are interested in designing novel 3D shape descriptors. However, the relationship between multiple views and point clouds needs to be further elucidated. We propose a multimodal method that combines the features of point clouds and multiple views, i.e., point-view relation neural network embedded with both attention mechanism and Radon transform, to obtain better descriptors. First, a two-dimensional linear Radon transform is performed to investigate linear and color features in multiple views, and the features are used as the input of our network to enable significant distinctions between different views. Moreover, a convolutional block attention module is adopted to enhance the features of point clouds and hence improve the expression ability of feature descriptors. The effectiveness of the proposed method is verified using ModelNet40 and ModelNet10 datasets. Experimental results show that our method can effectively improve the capability of feature extraction and expression as well as achieve state-of-the-art performance on two well-known 3D datasets.

Multi-View 3D Shape Recognition via Correspondence-Aware Deep Learning.

In recent years, multi-view learning has emerged as a promising approach for 3D shape recognition, which identifies a 3D shape based on its 2D views taken from different viewpoints. Usually, the correspondences inside a view or across different views encode the spatial arrangement of object parts and the symmetry of the object, which provide useful geometric cues for recognition. However, such view correspondences have not been explicitly and fully exploited in existing work. In this paper, we propose a correspondence-aware representation (CAR) module, which explicitly finds potential intra-view correspondences and cross-view correspondences via k NN search in semantic space and then aggregates the shape features from the correspondences via learned transforms. Particularly, the spatial relations of correspondences in terms of their viewpoint positions and intra-view locations are taken into account for learning correspondence-aware features. Incorporating the CAR module into a ResNet-18 backbone, we propose an effective deep model called CAR-Net for 3D shape classification and retrieval. Extensive experiments have demonstrated the effectiveness of the CAR module as well as the excellent performance of the CAR-Net.

PVCLN: Point-View Complementary Learning Network for 3D Shape Recognition

As an important topic in computer vision and multimedia analysis, 3D shape recognition has attracted much research attention in recent years. For point cloud data and multiview data, various approaches have been proposed with remarkable performance. However, few works simultaneously employ the point cloud data and multiview data to represent 3D shapes, which is complementary and beneficial in our consideration. Moreover, existing multimodal approaches mainly focus on the multimodal fusion strategy or on exploring the relation between them. However, the intra-modality characteristic information and inter-modality complementary information are ignored in these methods. In this paper, we tackle the above limitations by introducing a novel Point-View Complementary Learning Network (PVCLN) to explore the potential of both the complementary information and characteristic information for 3D shape recognition. Inspired by the successful application of graph neural networks in capturing relations between features, we introduce a novel multimodal fusion strategy. Concretely, we first separately extract the visual feature from multiview data and structural feature from point cloud data. We then project the visual feature and structural feature into the same feature space to learn the complementary information between two modalities by modeling the inter-modality affinities. The characteristic information in each modality is also preserved by considering the intra-modality affinities. The intra-modality and inter-modality affinities compensate for the lacking characteristic information and enhance the complementary information in the feature learning process. Finally, the updated visual and structural features are further combined to achieve a unified representation for a 3D shape. We conduct extensive experiments to validate the superiority of the overall network and the effectiveness of each component. The proposed method is evaluated on the ModelNet40 dataset and the experimental results demonstrate that our framework achieves competitive performance in the 3D shape recognition task.

Balanced principal component for 3D shape recognition using convolutional neural networks

Currently, PCA (principal component analysis) is widely used in many neural networks and has become a crucial part of the convolutional neural network (CNN) feature extraction. However, whether PCA is suitable for this process remains to be elucidated. The authors proposed a new method called balanced principal component (BPC) that generates a balanced local feature and combines with CNN as a layer to cope with the fusion problem. Specifically, BPC layer includes regionalisation module and average compression PCA (AC-PCA) module. First, they used regionalisation module to generate some sub-region that focuses on the local feature in each view. Secondly, the AC-PCA module is a computational process that enlarges the feature matrix by PCA and eventually compacts the matrix to a one-dimensional (1D) vector by AC. Next, all 1D vectors are compacted by AC to obtain a multi-dimensional balance. Finally, they designed this layer with an end-to-end trainable structure to promote the feature extraction task of CNN. They addressed 3D shapes using a projection method that is pre-trained on ImageNet and migration learning on ModelNet dataset. By comparing with the state-of-the-art network, they achieved a significant gain in performance of retrieval and classification tasks.

MMFN

In recent years, research into 3D shape recognition in the field of multimedia and computer vision has attracted wide attention. With the rapid development of deep learning, various deep models have achieved state-of-the-art performance based on different representations. There are many modalities for representing a 3D model, such as point cloud, multiview, and panorama view. Deep learning models based on these different modalities have different concerns, and all of them have achieved high performance for 3D shape recognition. However, all of these methods ignore the multimodality information in conditions where the same 3D model is represented by different modalities. Thus, we can obtain a better descriptor by guiding the training to consider these multiple representations. In this article, we propose MMFN, a novel multimodal fusion network for 3D shape recognition that employs correlations between the different modalities to generate a fused descriptor, which is more robust. In particular, we design two novel loss functions to help the model learn the correlation information during training. The first is correlation loss, which focuses on the correlations among different descriptors generated from different structures. This approach reduces the training time and improves the robustness of the fused descriptor of the 3D model. The second is instance loss, which preserves the independence of each modality and utilizes feature differentiation to guide model learning during the training process. More specifically, we use the weighted fusion method, which applies statistical methods to obtain robust descriptors that maximize the advantages of the information from the different modalities. We evaluated the proposed method on the ModelNet40 and ShapeNetCore55 datasets for 3D shape classification and retrieval tasks. The experimental results and comparisons with state-of-the-art methods demonstrate the superiority of our approach.

Semi‐Supervised 3D Shape Recognition via Multimodal Deep Co‐training

Abstract3D shape recognition has been actively investigated in the field of computer graphics. With the rapid development of deep learning, various deep models have been introduced and achieved remarkable results. Most 3D shape recognition methods are supervised and learn only from the large amount of labeled shapes. However, it is expensive and time consuming to obtain such a large training set. In contrast to these methods, this paper studies a semi‐supervised learning framework to train a deep model for 3D shape recognition by using both labeled and unlabeled shapes. Inspired by the co‐training algorithm, our method iterates between model training and pseudo‐label generation phases. In the model training phase, we train two deep networks based on the point cloud and multi‐view representation simultaneously. In the pseudo‐label generation phase, we generate the pseudo‐labels of the unlabeled shapes using the joint prediction of two networks, which augments the labeled set for the next iteration. To extract more reliable consensus information from multiple representations, we propose an uncertainty‐aware consistency loss function to combine the two networks into a multimodal network. This not only encourages the two networks to give similar predictions on the unlabeled set, but also eliminates the negative influence of the large performance gap between the two networks. Experiments on the benchmark ModelNet40 demonstrate that, with only 10% labeled training data, our approach achieves competitive performance to the results reported by supervised methods.

Deep learning for non-rigid 3D shape classification based on informative images

In order to enhance the discernment of features in view-based 3D shape recognition, we propose a joint convolutional neural network (CNN) learning model based on informative images. It learns deep features from intrinsic feature images and extrinsic 2D views, and generates a synthetic feature vector via weighted aggregation and refinement process, which has achieved remarkable improvement in non-rigid 3D shape classification. Our joint CNNs model contains three parts: the first part is the geometry-based feature generation unit. We provide a discriminative BoF (bag of features) image descriptor and construct CNN framework to learn the geometric features of the model. The second part is the view-based feature generation unit. We establish a parallel CNN to extract spatial features from optimized 2D views. The third part is a score generation and refinement unit, which automatically learns the weighted scores of geometric features and spatial features. Finally, the aggregated feature is refined in a CNN framework and serves as an informative shape descriptor for recognition task. The experimental results demonstrate that our deep features have the strong discerning ability. Thus, better performance and robustness can be obtained compared to state-of-the-art methods.