3D Shape Segmentation Research Articles

Multi-scale Knowledge Transfer Vision Transformer for 3D vessel shape segmentation

In order to facilitate the robust and precise 3D vessel shape extraction and quantification from in-vivo Magnetic Resonance Imaging (MRI), this paper presents a novel multi-scale Knowledge Transfer Vision Transformer (i.e., KT-ViT) for 3D vessel shape segmentation. First, it uniquely integrates convolutional embeddings with transformer in a U-net architecture, which simultaneously responds to local receptive fields with convolution layers and global contexts with transformer encoders in a multi-scale fashion. Therefore, it intrinsically enriches local vessel feature and simultaneously promotes global connectivity and continuity for a more accurate and reliable vessel shape segmentation. Furthermore, to enable using relatively low-resolution (LR) images to segment fine scale vessel shapes, a novel knowledge transfer network is designed to explore the inter-dependencies of data and automatically transfer the knowledge gained from high-resolution (HR) data to the low-resolution handling network at multiple levels, including the multi-scale feature levels and the decision level, through an integration of multi-level loss functions. The modeling capability of fine-scale vessel shape data distribution, possessed by the HR image transformer network, can be transferred to the LR image transformer to enhance its knowledge for fine vessel shape segmentation. Extensive experimental results on public image datasets have demonstrated that our method outperforms all other state-of-the-art deep learning methods.

Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objects.

The objective of this paper is to learn dense 3D shape correspondence for topology-varying generic objects in an unsupervised manner. Conventional implicit functions estimate the occupancy of a 3D point given a shape latent code. Instead, our novel implicit function produces a probabilistic embedding to represent each 3D point in a part embedding space. Assuming the corresponding points are similar in the embedding space, we implement dense correspondence through an inverse function mapping from the part embedding vector to a corresponded 3D point. Both functions are jointly learned with several effective and uncertainty-aware loss functions to realize our assumption, together with the encoder generating the shape latent code. During inference, if a user selects an arbitrary point on the source shape, our algorithm can automatically generate a confidence score indicating whether there is a correspondence on the target shape, as well as the corresponding semantic point if there is one. Such a mechanism inherently benefits man-made objects with different part constitutions. The effectiveness of our approach is demonstrated through unsupervised 3D semantic correspondence and shape segmentation.

Semi-Supervised 3D Shape Segmentation via Self Refining.

3D shape segmentation is a fundamental and crucial task in the field of image processing and 3D shape analysis. To segment 3D shapes using data-driven methods, a fully labeled dataset is usually required. However, obtaining such a dataset can be a daunting task, as manual face-level labeling is both time-consuming and labor-intensive. In this paper, we present a semi-supervised framework for 3D shape segmentation that uses a small, fully labeled set of 3D shapes, as well as a weakly labeled set of 3D shapes with sparse scribble labels. Our framework first employs an auxiliary network to generate initial fully labeled segmentation labels for the sparsely labeled dataset, which helps in training the primary network. During training, the self-refine module uses increasingly accurate predictions of the primary network to improve the labels generated by the auxiliary network. Our proposed method achieves better segmentation performance than previous semi-supervised methods, as demonstrated by extensive benchmark tests, while also performing comparably to supervised methods.

3D Shape Segmentation via Attentive Nonuniform Downsampling

3D Shape Segmentation via Attentive Nonuniform Downsampling

Cross‐Shape Attention for Part Segmentation of 3D Point Clouds

AbstractWe present a deep learning method that propagates point‐wise feature representations across shapes within a collection for the purpose of 3D shape segmentation. We propose a cross‐shape attention mechanism to enable interactions between a shape's point‐wise features and those of other shapes. The mechanism assesses both the degree of interaction between points and also mediates feature propagation across shapes, improving the accuracy and consistency of the resulting point‐wise feature representations for shape segmentation. Our method also proposes a shape retrieval measure to select suitable shapes for cross‐shape attention operations for each test shape. Our experiments demonstrate that our approach yields state‐of‐the‐art results in the popular PartNet dataset.

Mesh Convolutional Networks With Face and Vertex Feature Operators.

Deep learning techniques have proven effective in many applications, but these implementations mostly apply to data in one or two dimensions. Handling 3D data is more challenging due to its irregularity and complexity, and there is a growing interest in adapting deep learning techniques to the 3D domain. A recent successful approach called MeshCNN consists of a set of convolutional and pooling operators applied to the edges of triangular meshes. While this approach produced superb results in classification and segmentation of 3D shapes, it can only be applied to edges of a mesh, which can constitute a disadvantage for applications where the focuses are other primitives of the mesh. In this study, we propose face-based and vertex-based operators for mesh convolutional networks. We design two novel architectures based on the MeshCNN network that can operate on faces and vertices of a mesh, respectively. We demonstrate that the proposed face-based architecture outperforms the original MeshCNN implementation in mesh classification and mesh segmentation, setting the new state of the art on benchmark datasets. In addition, we extend the vertex-based operator to fit in the Point2Mesh model for mesh reconstruction from clean, noisy, and incomplete point clouds. While no statistically significant performance improvements are observed, the model training and inference time are reduced by the proposed approach by 91% and 20%, respectively, as compared with the original Point2Mesh model.

Semi-supervised 3D shape segmentation with multilevel consistency and part substitution

The lack of fine-grained 3D shape segmentation data is the main obstacle to developing learning-based 3D segmentation techniques. We propose an effective semi-supervised method for learning 3D segmentations from a few labeled 3D shapes and a large amount of unlabeled 3D data. For the unlabeled data, we present a novel multilevel consistency loss to enforce consistency of network predictions between perturbed copies of a 3D shape at multiple levels: point level, part level, and hierarchical level. For the labeled data, we develop a simple yet effective part substitution scheme to augment the labeled 3D shapes with more structural variations to enhance training. Our method has been extensively validated on the task of 3D object semantic segmentation on PartNet and ShapeNetPart, and indoor scene semantic segmentation on ScanNet. It exhibits superior performance to existing semi-supervised and unsupervised pre-training 3D approaches.

3D Shape Segmentation: A Review

Background: Shape segmentation is commonly required in many engineering fields to separate a 3D shape into pieces for some specific applications. Although there are different methods proposed to segment the 3D shape, there is a lack of analyses of their efficiency and accuracy. It is a challenge to select an effective method to meet a particular requirement of the shape segmentation. Objective: This paper reviews existing methods of the shape segmentation to summarize the methods and processes to identify their pros and cons. Method: The process of the shape segmentation is summarized in two steps of the feature extraction and model separation. Results: Shape features are identified from the available methods. Different methods of the shape segmentation are evaluated. The challenge and trend of the shape segmentation are discussed. Conclusion: Clustering is the most used method for the shape segmentation. Machine learning methods are trend of 3D shape segmentations for identification, analysis and reconstruction of large-scale models.

One Sketch for All: One-Shot Personalized Sketch Segmentation.

We present the first one-shot personalized sketch segmentation method. We aim to segment all sketches belonging to the same category provisioned with a single sketch with a given part annotation while (i) preserving the parts semantics embedded in the exemplar, and (ii) being robust to input style and abstraction. We refer to this scenario as personalized. With that, we importantly enable a much-desired personalization capability for downstream fine-grained sketch analysis tasks. To train a robust segmentation module, we deform the exemplar sketch to each of the available sketches of the same category. Our method generalizes to sketches not observed during training. Our central contribution is a sketch-specific hierarchical deformation network. Given a multi-level sketch-strokes encoding obtained via a graph convolutional network, our method estimates rigid-body transformation from the target to the exemplar, on the upper level. Finer deformation from the exemplar to the globally warped target sketch is further obtained through stroke-wise deformations, on the lower-level. Both levels of deformation are guided by mean squared distances between the keypoints learned without supervision, ensuring that the stroke semantics are preserved. We evaluate our method against the state-of-the-art segmentation and perceptual grouping baselines re-purposed for the one-shot setting and against two few-shot 3D shape segmentation methods. We show that our method outperforms all the alternatives by more than 10% on average. Ablation studies further demonstrate that our method is robust to personalization: changes in input part semantics and style differences.

3D Shape Segmentation Using Soft Density Peak Clustering and Semi-Supervised Learning

Shape segmentation plays a vital role in shape analysis. Recent research uses fully-supervised learning methods to achieve state-of-the-art performance. However, acquiring fully-labeled training data is usually an extremely expensive process. In this paper, we present a novel semi-supervised algorithm for 3D shape segmentation. In our method, users are only required to locate several seed faces with a very simple interaction by using our soft density peak clustering method. Our method can automatically learn the required label information and produce satisfactory segmentation results with our novel optimization model. Various experimental results show that the presented method can achieve superior segmentation performance over previous unsupervised methods and comparable performance to the fully-supervised methods.

A Magic Wand Selection Tool for Surface of 3D Model

Introduction: segmentation of 3d shapes is a fundamental problem in computer graphics and computer-aided design. It has received much plays attention in recent years. The analysis and research methods of 3d mesh models have established reliable mathematical foundations in graphics and geometric modeling. Compared with color and texture, shape features describe the shape information of objects from geometric structure features. And it an important role in a wide range of applications, including mesh parameterization, skeleton extraction, resolution modeling, shape retrieval, character recognitio,, robot navigation, and many others. Methods: The interactive selection surface of models is mainly used for shape segmentation. The common method is boundary-based selection, which requires user input some stokes near the edge of the selected or segmented region. Chen et al.introduced an approach to join the specified points form the boundaries for region segmentation on the surface. Funkhouser et al. improve the Dijkstra algorithm to find segmentation boundary contour. The graph cut algorithm use the distance between the surface and its convex hull as the growing criteria to decompose a shape into meaningful components. The watershed algorithm, widely used for image segmentation, is a region-growing algorithm with multiple seed points. Wu and Levine use simulated electrical charge distributions over the mesh to deal with the 3D part segmentation problem. Other methods using a watershed algorithm for surface decomposition. Results: We implemented our algorithm in C++ and Open MP and conducted the experiments on a PC with a 3.07 GHz Intel(R) Core(TM) i7 CPU and 6 GB memory. Our method can get a similar region under different interaction vertices in specific regions. Figure 6a and Figure 6b are the calculation results of tolerance region selection of this algorithm in a certain region of kitten model at two different interaction points, from which we can see the obtained regions are similar from different vertices in this region. Figure 6c and figure 6d are two different interactive points in the same region, and the region selection results are obtained by Region growing technique. Discussion: In this paper, we proposed a novel magic wand selection tool to interactive select surface of 3D model. The feature vector is constructed by extracting the HKS feature descriptor and means curvature of 3D model surface, which allows users to input the feature tolerance value for region selection and improves the self-interaction of users. Many experiments show that our algorithm has obvious advantages in speed and effectiveness. The interactive generation of region boundary is very useful for many applications including model segmentation. Conclusion: In consideration of a couple of requirements including user-friendliness and effectiveness in model region selection, we propose a novel magic wand selection tool to interactive selection surface of 3D models. First, we pre-compute the heat kernel feature and mean curvature of the surface, and then form the eigenvector of the model. Then we provide two ways for region selection. One is to select the region according to the feature of tolerance value. The other is to select the region that aligns with stroke automatically. Finally, we use the geometry optimization approach to improve the performance of the computing region con-tours. Extensive experimental results show that our algorithm is efficient and effective.

3D Dental model segmentation with graph attentional convolution network

Precisely segmenting teeth from digitized 3D dental models is an essential task in computer-aided orthodontic surgical planning. In recent years, various deep learning-based methods have been proposed to process dental models for teeth segmentation, however, these methods usually ignore or coarsely model the dependency between vertices/mesh cells in local space, which fails to exploit local geometric details that are critical to capture complete teeth structure. In this paper, we propose a specific end-to-end network for teeth segmentation on 3D dental models. By constructing a graph for the raw mesh data, our network adopts a series of graph attentional convolution layers and a global structure branch to extract fine-grained local geometric feature and global feature, respectively. Subsequently, these two features are further fused to learn comprehensive information for cell-wise segmentation tasks. We have evaluated our network on a real-patient dataset of dental models acquired through 3D intraoral scanners, and experimental results show that our method outperforms state-of-the-art deep learning methods for 3D shape segmentation.

FlyBy CNN: A 3D surface segmentation framework.

In this paper, we present FlyBy CNN, a novel deep learning based approach for 3D shape segmentation. FlyByCNN consists of sampling the surface of the 3D object from different view points and extracting surface features such as the normal vectors. The generated 2D images are then analyzed via 2D convolutional neural networks such as RUNETs. We test our framework in a dental application for segmentation of intra-oral surfaces. The RUNET is trained for the segmentation task using image pairs of surface features and image labels as ground truth. The resulting labels from each segmented image are put back into the surface thanks to our sampling approach that generates 1-1 correspondence of image pixels and triangles in the surface model. The segmentation task achieved an accuracy of 0.9.

CAD Model Segmentation Via Deep Learning

Computer aided design (CAD) models are widely employed in the current computer aided engineering or finite element analysis (FEA) systems that necessitate an optimal meshing as a function of their geometry. To this effect, the sub-mapping method is advantageous, as it segments the CAD model into different sub-parts, with the aim mesh them independently. Many of the existing 3D shape segmentation methods in literature are not suited to CAD models. Therefore, we propose a novel approach for the segmentation of CAD models by harnessing deep learning technologies. First, we refined the model and extracted local geometric features from its shape. Subsequently, we devised a convolutional neural network (CNN)-inspired neural network trained with a custom dataset. Experimental results demonstrate the robustness of our approach and its potential to adapt to augmented datasets in future.

AUTOMATED PART DECOMPOSITION FOR PRODUCT ARCHITECTURE MODELING

AbstractThis paper presents an algorithm that contributes to an automatic decomposition of a mechanical part based on geometric features and methods of unsupervised machine learning. For the development of the algorithm, existing techniques of 3D shape segmentation, especially surface-based part segmentation procedures are reviewed and important areas of activities are revealed. The developed multi-step approach results in an abstract product model. This representation leads to a new way of designing and redesigning parts for the novel hybrid manufacturing concept Incremental Manufacturing (IM).

Deep Multi-Scale Mesh Feature Learning for Automated Labeling of Raw Dental Surfaces From 3D Intraoral Scanners.

Precisely labeling teeth on digitalized 3D dental surface models is the precondition for tooth position rearrangements in orthodontic treatment planning. However, it is a challenging task primarily due to the abnormal and varying appearance of patients' teeth. The emerging utilization of intraoral scanners (IOSs) in clinics further increases the difficulty in automated tooth labeling, as the raw surfaces acquired by IOS are typically low-quality at gingival and deep intraoral regions. In recent years, some pioneering end-to-end methods (e.g., PointNet) have been proposed in the communities of computer vision and graphics to consume directly raw surface for 3D shape segmentation. Although these methods are potentially applicable to our task, most of them fail to capture fine-grained local geometric context that is critical to the identification of small teeth with varying shapes and appearances. In this paper, we propose an end-to-end deep-learning method, called MeshSegNet, for automated tooth labeling on raw dental surfaces. Using multiple raw surface attributes as inputs, MeshSegNet integrates a series of graph-constrained learning modules along its forward path to hierarchically extract multi-scale local contextual features. Then, a dense fusion strategy is applied to combine local-to-global geometric features for the learning of higher-level features for mesh cell annotation. The predictions produced by our MeshSegNet are further post-processed by a graph-cut refinement step for final segmentation. We evaluated MeshSegNet using a real-patient dataset consisting of raw maxillary surfaces acquired by 3D IOS. Experimental results, performed 5-fold cross-validation, demonstrate that MeshSegNet significantly outperforms state-of-the-art deep learning methods for 3D shape segmentation.

SEG-MAT: 3D Shape Segmentation Using Medial Axis Transform.

Segmenting arbitrary 3D objects into constituent parts that are structurally meaningful is a fundamental problem encountered in a wide range of computer graphics applications. Existing methods for 3D shape segmentation suffer from complex geometry processing and heavy computation caused by using low-level features and fragmented segmentation results due to the lack of global consideration. We present an efficient method, called SEG-MAT, based on the medial axis transform (MAT) of the input shape. Specifically, with the rich geometrical and structural information encoded in the MAT, we are able to develop a simple and principled approach to effectively identify the various types of junctions between different parts of a 3D shape. Extensive evaluations and comparisons show that our method outperforms the state-of-the-art methods in terms of segmentation quality and is also one order of magnitude faster.

Label transfer between images and 3D shapes via local correspondence encoding

We present a generic and accurate method to transfer part-level labels from annotated 3D shapes to images or annotated images to 3D shapes. The label transfer problem is formulated as a Conditional random field (CRF) model, in which a novel local correspondence encoding term encoding the probability of label assignment of each pixel as a label histogram serves as the likelihood of the labels. With the aid of local correspondence encoding, highly accurate part-level label transfer results can be easily achieved. Our method works well for the bi-directional label transfer between 3D shapes and images, suitable to image-guided 3D shape segmentation, 3D shape-guided image segmentation, and has also found various applications. We thoroughly evaluate our method and demonstrate its superiority over the state-of-the-art methods through experiments on benchmark datasets.

Scribble-Based 3D Shape Segmentation via Weakly-Supervised Learning.

Shape segmentation is a fundamental problem in shape analysis. Previous research shows that prior knowledge helps to improve the segmentation accuracy and quality. However, completely labeling each 3D shape in a large training data set requires a heavy manual workload. In this paper, we propose a novel weakly-supervised algorithm for segmenting 3D shapes using deep learning. Our method jointly propagates information from scribbles to unlabeled faces and learns deep neural network parameters. Therefore, it does not rely on completely labeled training shapes and only needs a really simple and convenient scribble-based partially labeling process, instead of the extremely time-consuming and tedious fully labeling processes. Various experimental results demonstrate the proposed method's superior segmentation performance over the previous unsupervised approaches and comparable segmentation performance to the state-of-the-art fully supervised methods.

Corrigendum to ``3D shape segmentation via shape fully convolutional networks'' [Computers & Graphics 70 (2018) 128–139

Corrigendum to ``3D shape segmentation via shape fully convolutional networks'' [Computers & Graphics 70 (2018) 128–139