3D Shape Representation Research Articles

Reconstructing Geometric Models from the Point Clouds of Buildings Based on a Transformer Network

Geometric building models are essential in BIM technology. The reconstruction results using current methods are usually represented using mesh, which is limited to visualization purposes and hard to directly import into BIM or modeling software for further application. In this paper, we propose a building model reconstruction method based on a transformer network (DeepBuilding). Instead of reconstructing the polyhedron model of buildings, we strive to recover the CAD modeling operation of constructing the building models from the building point cloud. By representing the building model with its modeling sequence, the reconstruction results can be imported into BIM software for further application. We first translate the procedure of constructing a building model into a command sequence that can be vectorized and processed by the transformer network. Then, we propose a transformer-based network that can convert input point clouds into the vectorized representation of the modeling sequences by decoding the geometry information encoded in the point features. A tool is developed to convert the vectorized modeling sequence into a 3D shape representation (such as mesh) or file format that other BIM software supports. Comprehensive experiments are conducted, and the evaluation results demonstrate that our method can produce competitive reconstruction results with high geometric fidelity while preserving more details of the building reconstruction.

Learning Meshing from Delaunay Triangulation for 3D Shape Representation

Learning Meshing from Delaunay Triangulation for 3D Shape Representation

RJAN: Region‐based joint attention network for 3D shape recognition

AbstractAs an essential field of multimedia and computer vision, 3D shape recognition has attracted much research attention in recent years. Multiview‐based approaches have demonstrated their superiority in generating effective 3D shape representations. Typical methods usually extract the multiview global features and aggregate them together to generate 3D shape descriptors. However, there exist two disadvantages: First, the mainstream methods ignore the comprehensive exploration of local information in each view. Second, many approaches roughly aggregate multiview features by adding or concatenating them together. The information loss for some discriminative characteristics limits the representation effectiveness. To address these problems, a novel architecture named region‐based joint attention network (RJAN) was proposed. Specifically, the authors first design a hierarchical local information exploration module for view descriptor extraction. The region‐to‐region and channel‐to‐channel relationships from different granularities can be comprehensively explored and utilised to provide more discriminative characteristics for view feature learning. Subsequently, a novel relation‐aware view aggregation module is designed to aggregate the multiview features for shape descriptor generation, considering the view‐to‐view relationships. Extensive experiments were conducted on three public databases: ModelNet40, ModelNet10, and ShapeNetCore55. RJAN achieves state‐of‐the‐art performance in the tasks of 3D shape classification and 3D shape retrieval, which demonstrates the effectiveness of RJAN. The code has been released on https://github.com/slurrpp/RJAN.

Learning intrinsic shape representations via spectral mesh convolutions

We introduce spectral-based convolutional operators embedded within Generalized Graph Neural Networks (G-GNNs). These operators enable deep learning on graphs through a learnable, energy-driven evolution process. This approach empowers us to impose specific properties on the graph convolutional kernel directly derived from the corresponding variational formulations. Our model incorporates both parameterized and non-parameterized graph Laplacian-based energies within the generalized graph convolutional layer to address features like smoothness, sharpness, and compact support. By making appropriate choices within our G-GNN framework, we pave the way for designing novel paradigms for 3D shape representation, reconstruction, and processing, while also enabling effective feature embeddings for intrinsic neural fields.

Data-Driven Car Drag Prediction With Depth and Normal Renderings

Abstract Generative artificial intelligence (AI) models have made significant progress in automating the creation of 3D shapes, which has the potential to transform car design. In engineering design and optimization, evaluating engineering metrics is crucial. To make generative models performance-aware and enable them to create high-performing designs, surrogate modeling of these metrics is necessary. However, the currently used representations of 3D shapes either require extensive computational resources to learn or suffer from significant information loss, which impairs their effectiveness in surrogate modeling. To address this issue, we propose a new 2D representation of 3D shapes. We develop a surrogate drag model based on this representation to verify its effectiveness in predicting 3D car drag. We construct a diverse dataset of 4535 high-quality 3D car meshes labeled by drag coefficients computed from computational fluid dynamics simulations to train our model. Our experiments demonstrate that our model can accurately and efficiently evaluate drag coefficients with an R2 value above 0.84 for various car categories. Our model is implemented using deep neural networks, making it compatible with recent AI image generation tools (such as stable diffusion) and a significant step toward the automatic generation of drag-optimized car designs. Moreover, we demonstrate a case study using the proposed surrogate model to guide a diffusion-based deep generative model for drag-optimized car body synthesis.

LISR: Learning Linear 3D Implicit Surface Representation Using Compactly Supported Radial Basis Functions

Implicit 3D surface reconstruction of an object from its partial and noisy 3D point cloud scan is the classical geometry processing and 3D computer vision problem. In the literature, various 3D shape representations have been developed, differing in memory efficiency and shape retrieval effectiveness, such as volumetric, parametric, and implicit surfaces. Radial basis functions provide memory-efficient parameterization of the implicit surface. However, we show that training a neural network using the mean squared error between the ground-truth implicit surface and the linear basis-based implicit surfaces does not converge to the global solution. In this work, we propose locally supported compact radial basis functions for a linear representation of the implicit surface. This representation enables us to generate 3D shapes with arbitrary topologies at any resolution due to their continuous nature. We then propose a neural network architecture for learning the linear implicit shape representation of the 3D surface of an object. We learn linear implicit shapes within a supervised learning framework using ground truth Signed-Distance Field (SDF) data for guidance. The classical strategies face difficulties in finding linear implicit shapes from a given 3D point cloud due to numerical issues (requires solving inverse of a large matrix) in basis and query point selection. The proposed approach achieves better Chamfer distance and comparable F-score than the state-of-the-art approach on the benchmark dataset. We also show the effectiveness of the proposed approach by using it for the 3D shape completion task.

Learning Spatially Collaged Fourier Bases for Implicit Neural Representation

Existing approaches to Implicit Neural Representation (INR) can be interpreted as a global scene representation via a linear combination of Fourier bases of different frequencies. However, such universal basis functions can limit the representation capability in local regions where a specific component is unnecessary, resulting in unpleasant artifacts. To this end, we introduce a learnable spatial mask that effectively dispatches distinct Fourier bases into respective regions. This translates into collaging Fourier patches, thus enabling an accurate representation of complex signals. Comprehensive experiments demonstrate the superior reconstruction quality of the proposed approach over existing baselines across various INR tasks, including image fitting, video representation, and 3D shape representation. Our method outperforms all other baselines, improving the image fitting PSNR by over 3dB and 3D reconstruction to 98.81 IoU and 0.0011 Chamfer Distance.

CROMOSim: A Deep Learning-Based Cross-Modality Inertial Measurement Simulator

With the prevalence of wearable devices, inertial measurement unit (IMU) data has been utilized in monitoring and assessing human mobility such as human activity recognition (HAR) and human pose estimation (HPE). Training deep neural network (DNN) models for these tasks require a large amount of labelled data, which are hard to acquire in uncontrolled environments. To mitigate the data scarcity problem, we design CROMOSim, a cross-modality sensor simulator that simulates high fidelity virtual IMU sensor data from motion capture systems or monocular RGB cameras. It utilizes a skinned multi-person linear model (SMPL) for 3D body pose and shape representations to enable simulation from arbitrary on-body positions. Then a DNN model is trained to learn the functional mapping from imperfect trajectory estimations in a 3D SMPL body tri-mesh due to measurement noise, calibration errors, occlusion and other modelling artifacts, to IMU data. We evaluate the fidelity of CROMOSim simulated data and its utility in data augmentation on various HAR and HPE datasets. Extensive empirical results show that the proposed model achieves a 6.7% improvement over baseline methods in a HAR task.

3D Shape Contrastive Representation Learning with Adversarial Examples

Current supervised methods for 3D shape representation learning have achieved satisfying performance, yet require extensive human-labeled datasets. Unsupervised learning-based methods provide a viable solution by learning shape representations without using ground truth labels. In this study, we develop a contrastive learning framework for unsupervised representation learning of 3D shapes. Specifically, in order to encourage models to pay more attention to useful information during representation learning, we first introduce a new paradigm for critical points search based on the adversarial mechanism. We extract critical points with a larger impact on the global feature by attacking a pre-trained auto-encoder model, and apply data augmentations on these points to generate adversarial examples. Taking a pair of adversarial examples as inputs, we obtain their intermediate embeddings and global representations of corresponding inputs, which are then transformed into latent spaces by two predictor heads. Finally, we train the proposed model by maximizing the agreements on these latent spaces via Normalized Temperature-scaled Cross Entropy (NT-Xent) loss and a newly designed Cross-layer Normalized Temperature-scaled Cross Entropy (Cross-NT-Xent) loss, where the latter is proposed in this paper to enforce cross-layer feature similarities. The effectiveness, robustness, and transferability of learned representations are validated on three downstream tasks, including object classification, few-shot classification, and shape retrieval. Experiments on three benchmark datasets show that our learned representations achieve better or competitive performance than current state-of-the-art methods in these downstream tasks. Moreover, our model can easily be extended to 3D part segmentation and scene segmentation tasks.

Shared Latent Membership Enables Joint Shape Abstraction and Segmentation with Deformable Superquadrics.

Part-level 3D shape representations are crucial to shape reasoning and understanding. Two key sub-tasks are (i) shape abstraction, creating primitive-based object parts, and (ii) shape segmentation, finding partition-based object parts. However, for 3D object point clouds, most advanced methods produce parts relying on task-specific priors, such as similarity metrics and primitive geometries, resulting in misleading parts that deviate from semantics. To address prior limitations, we establish a foundation for joint shape abstraction and shape segmentation as formal linear transformations within a shared latent space, encapsulating essential dual-purpose membership information linking points and object parts for mutual reinforcement. We demonstrate that the transformations are underpinned by a derivation based on k-means, Non-negative Matrix Factorization (NMF), and the attention mechanism. As a result, we introduce Latent Membership Pursuit (LMP) for joint optimization of shape abstraction and segmentation. LMP utilizes a shared latent representation of object part membership to autonomously identify common object parts in both tasks without any supervision and priors. Furthermore, we adapt deformable superquadrics (DSQs) for primitives to capture variable part-level geometric and semantic information. Experiments on benchmark datasets validate that our approach enables mutual learning of shape abstraction and segmentation, and promotes consistent interpretations of 3D object shapes across instances and even categories in a fully unsupervised manner.

Brep2Seq: a dataset and hierarchical deep learning network for reconstruction and generation of computer-aided design models

Abstract Three-dimensional (3D) reconstruction is a significant research topic in the field of computer-aided design (CAD), which is used to recover editable CAD models from original shapes, including point clouds, voxels, meshes, and boundary representations (B-rep). Recently, there has been considerable research interest in deep model generation due to the increasing potential of deep learning methods. To address the challenges of 3D reconstruction and generation, we propose Brep2Seq, a novel deep neural network designed to transform the B-rep model into a sequence of editable parametrized feature-based modeling operations comprising principal primitives and detailed features. Brep2Seq employs an encoder-decoder architecture based on the transformer, leveraging geometry and topological information within B-rep models to extract the feature representation of the original 3D shape. Due to its hierarchical network architecture and training strategy, Brep2Seq achieved improved model reconstruction and controllable model generation by distinguishing between the primary shape and detailed features of CAD models. To train Brep2Seq, a large-scale dataset comprising 1 million CAD designs is established through an automatic geometry synthesis method. Extensive experiments on both DeepCAD and Fusion 360 datasets demonstrate the effectiveness of Brep2Seq, and show its applicability to simple mechanical components in real-world scenarios. We further apply Brep2Seq to various downstream applications, including point cloud reconstruction, model interpolation, shape constraint generation, and CAD feature recognition.

Children Perception of 3D Usual Shapes in Virtual Environments

Improving the understanding of the 3D perception spatial ability for children is an important issue for research in the areas of cognitive and developmental psychology, particularly in virtual environments where perceptive behaviors could differ from real environment. The aim of this research is to evaluate if children from 6 to 10 years old are able to perceive usual 3D shapes displayed in virtual environment like in 3D world.In an experimentation, children (N =218) are split in groups faced with different modalities of representation of 3D shapes they are allowed to manipulate or only observe: virtual representations on touchpad or real representation with 3D models. They have to solve recognition task of usual 3D shapes (cube, cylinder, triangular prism, etc.). If results indicate that majority of children adequately perceived virtual usual 3D shapes, this ability is not acquired for everyone and is more problematic than perception of 3D models. Some shapes, like the triangular prism, were less adequately perceived. Results call for prudence regarding the use of virtual 3D shapes with children for example during 3D geometry learning.

EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape Generation

This paper presents a new text-guided technique for generating 3D shapes. The technique leverages a hybrid 3D shape representation, namely EXIM, combining the strengths of explicit and implicit representations. Specifically, the explicit stage controls the topology of the generated 3D shapes and enables local modifications, whereas the implicit stage refines the shape and paints it with plausible colors. Also, the hybrid approach separates the shape and color and generates color conditioned on shape to ensure shape-color consistency. Unlike the existing state-of-the-art methods, we achieve high-fidelity shape generation from natural-language descriptions without the need for time-consuming per-shape optimization or reliance on human-annotated texts during training or test-time optimization. Further, we demonstrate the applicability of our approach to generate indoor scenes with consistent styles using text-induced 3D shapes. Through extensive experiments, we demonstrate the compelling quality of our results and the high coherency of our generated shapes with the input texts, surpassing the performance of existing methods by a significant margin. Codes and models are released at https://github.com/liuzhengzhe/EXIM.

Comparing Shape Representations for the Aesthetic Judgement of 3D Shape Pairs.

Visual aesthetics of 3D shapes is a fundamental perceptual attribute. In this paper, we explore the question of how different shape representations affect the aesthetic judgments of shape pairs. Specifically, we compare human responses to perceptual aesthetics judgements on 3D shapes in pairs presented in different 3D shape representation such as voxels, points, wireframe, and polygon. In contrast to our own previous work [8] that explores this question for a few shape classes, this paper analyzes a larger number of shape classes. Our key finding is that human aesthetics judgements on relatively low resolution of points or voxels are comparable to polygon meshes, which implies that humans can often make their aesthetics decisions based on relatively coarse representations of the shapes. Our results have implications towards the data collection process of pairwise aesthetics data and the further use of such data in shape aesthetics and 3D modeling problems.

A robust approach to 3D neuron shape representation for quantification and classification

We consider the problem of finding an accurate representation of neuron shapes, extracting sub-cellular features, and classifying neurons based on neuron shapes. In neuroscience research, the skeleton representation is often used as a compact and abstract representation of neuron shapes. However, existing methods are limited to getting and analyzing “curve” skeletons which can only be applied for tubular shapes. This paper presents a 3D neuron morphology analysis method for more general and complex neuron shapes. First, we introduce the concept of skeleton mesh to represent general neuron shapes and propose a novel method for computing mesh representations from 3D surface point clouds. A skeleton graph is then obtained from skeleton mesh and is used to extract sub-cellular features. Finally, an unsupervised learning method is used to embed the skeleton graph for neuron classification. Extensive experiment results are provided and demonstrate the robustness of our method to analyze neuron morphology.

Point2MM: Learning medial mesh from point clouds

Medial mesh is the most commonly used representation of medial axis transform (MAT) which is a high-fidelity compact representation of 3D shape. Existing learning methods of extracting medial mesh from point clouds either have strict requirements of supervising training data, or are incapable of learning the complete form of medial mesh, leading to large reconstruction error. We introduce Point2MM, an unsupervised method for learning a complete medial mesh from point clouds. Our key idea is to use the envelop geometry of medial primitives – spheres, cones, and slabs – to capture the intrinsic geometry of shape, and the connectivity of medial mesh to capture the topology of shape. We firstly predict initial medial spheres by learning the geometric transformation of point clouds, then construct an initial connectivity of the medial spheres by learning the probability of medial cones and medial slabs with a novel unsupervised formulation. Finally we propose an iterative strategy for fine-tuning medial primitives. Extensive evaluations and comparisons show our method has superior accuracy and robustness in learning medial mesh from point clouds. In addition, the excessive training time is also a concern for our research, and it is a limitation where we need to make improvements in our future work.

MWFormer: Mesh Understanding with Window-based Transformer

Polygonal mesh has been proven to be a powerful representation of 3D shapes, given its efficiency in expressing shape surface while maintaining geometric and topological information. Increasing efforts have been made to design elaborate deep convolutional neural networks for meshes. However, these methods naturally ignore the global connectivity among mesh primitives due to the locality nature of convolution operations. In this paper, we introduce a transformer-like self-attention mechanism with down-sampling architectures for mesh learning to capture both the global and local relationships among mesh faces. To achieve this, we propose BFS-Pooling, which can convert a connected mesh into discrete tokens (i.e., a set of adjacent faces) with breath-first-search (BFS) and naturally build hierarchical architectures for mesh learning by pooling mesh tokens. Benefiting from BFS-Pooling, we design a hierarchical transformer architecture with a window-based local attention mechanism, Mesh Window Transformer (MWFormer). Experimental results demonstrate that MWFormer achieves the best or competitive performance in both mesh classification and mesh segmentation tasks. Code will be available.

TPNet: A novel mesh analysis method via topology preservation and perception enhancement

3D polygon mesh is an important and popular representation of 3D shapes in the field of computer graphics and computer-aided design. Recent works have introduced deep neural networks for mesh data analysis. However, it remains a great challenge for convolutional neural networks to learn mesh shape effectively due to the difficulty of deriving long-range information from the irregular data structure. Another tricky problem is how to explore a high effective representation of the input feature for mesh learning network. In this paper, we propose a novel 3D mesh learning method, named TPNet, which enhances the perception and learns long-range information by customizing dilated convolution for non-uniform mesh data. Specifically, we devise a faithful strategy that accurately locates the position where dilated convolutions can be adopted, despite the non-uniformity and irregularity of the mesh data. Furthermore, our method proposes a topology-preserved 7-dimension feature representation for mesh data and aggregates the features via stacks of convolution layers and dilated convolution layers. Extensive experiments demonstrate the effectiveness of our approach on 3D mesh learning tasks, where we show superior or at least comparable performance to the SOTA approaches.

ResAttn-recon: Residual self-attention based cortical surface reconstruction

Introduction: The accurate cerebral cortex surface reconstruction is crucial for the study of neurodegenerative diseases. Existing voxelwise segmentation-based approaches like FreeSurfer and FastSurfer are limited by the partial volume effect, meaning that reconstruction details highly rely on the resolution of the input volume. In the computer version area, the signed distance function has become an efficient method for 3D shape representation, the inherent continuous nature makes it easy to capture the fine details of the target object at an arbitrary resolution. Additionally, as one of the most valuable breakthroughs in deep learning research, attention is a powerful mechanism developed to enhance the performance of the encoder-decoder architecture.Methods: To further improve the reconstruction accuracy of the cortical surface, we proposed ResAttn-Recon, a residual self-attention based encoder-decoder framework. In this framework, we also developed a lightweight decoder network with skip connections. Furthermore, a truncated and weighted L1 loss function are proposed to accelerate network convergence, compared to simply applying the L1 loss function.Results: The intersection over union curve in the training process achieved a steeper slope and a higher peak (0.948 vs. 0.920) with a truncated L1 loss. Thus, the average symmetric surface distance (AD) for the inner and outer surfaces is 0.253 ± 0.051 and the average Hausdorff distance (HD) is 0.629 ± 0.186, which is lower than that of DeepCSR, whose absolute distance equals 0.283 ± 0.059 and Hausdorff distance equals 0.746 ± 0.245.Discussion: In conclusion, the proposed residual self-attention-based framework can be a promising approach for improving the cortical surface reconstruction performance.

Self-Supervised Auxiliary Domain Alignment for Unsupervised 2D Image-Based 3D Shape Retrieval

Unsupervised 2D image-based 3D shape retrieval aims to match the similar 3D unlabeled shapes when given a 2D labeled sample. Although a lot of methods have made a certain degree of progress, the performance of this task is still restricted due to the lack of target labels resulting in tremendous domain gap. In this paper, we aim to explore the discriminative representation of the unlabeled target 3D shapes and facilitate the procedure of domain adaptation by taking full advantage of multi-view information. To achieve the above goals, we propose an effective self-supervised auxiliary domain alignment (SADA) for unsupervised 2D image-based 3D shape retrieval. SADA mainly contains multi-view guided self-supervised feature learning and two auxiliary domain alignments, including intermediate domain alignment and multi-domain alignment. Firstly, we group multiple views of each 3D shape into two sub-target domains based on the view similarities and regard each other as the constraint to optimize the feature learning in an unsupervised manner. To reduce the difficulty of directly aligning the domain discrepancy, we combine the source labeled samples and target samples (pseudo labels) with the same category to generate an intermediate domain, which translates the source-target alignment into source-intermediate and intermediate-target alignments. Moreover, to explore the inner characteristics of target 3D shapes and provide more clues for better adaptation, multi-domain alignment is proposed to convert the source and single target domain alignment to the source and multiple target domain (one target domain and two sub-target domains) alignments. The adversarial training and semantic alignment are employed to fully excavate the relations between source domain and multiple target domains. Experiments on two challenging datasets show that the proposed method achieves competing performance in the unsupervised 2D image-based 3D shape retrieval task.