Non-rigid Shape Research Articles

Diverse Bases for Functional Spaces for Non-Rigid Shape Correspondence

Diverse Bases for Functional Spaces for Non-Rigid Shape Correspondence

Relation Constrained Capsule Graph Neural Networks for Non-rigid Shape Correspondence

Relation Constrained Capsule Graph Neural Networks for Non-rigid Shape Correspondence

Perceptual Biases in the Interpretation of Non-Rigid Shape Transformations from Motion.

Most existing research on the perception of 3D shape from motion has focused on rigidly moving objects. However, many natural objects deform non-rigidly, leading to image motion with no rigid interpretation. We investigated potential biases underlying the perception of non-rigid shape interpretations from motion. We presented observers with stimuli that were consistent with two qualitatively different interpretations. Observers were shown a two-part 3D object with the smaller part changing in length dynamically as the whole object rotated back and forth. In two experiments, we studied the misperception (i.e., perceptual reinterpretation) of the non-rigid length change to a part. In Experiment 1, observers misperceived this length change as a part orientation change (i.e., the smaller part was seen as articulating with respect to the larger part). In Experiment 2, the stimuli were similar, except the silhouette of the part was visible in the image. Here, the non-rigid length change was reinterpreted as a rigidly attached part with an "illusory" non-orthogonal horizontal angle relative to the larger part. We developed a model that incorporated this perceptual reinterpretation and could predict observer data. We propose that the visual system may be biased towards part-wise rigid interpretations of non-rigid motion, likely due to the ecological significance of movements of humans and other animals, which are generally constrained to move approximately part-wise rigidly. That is, not all non-rigid deformations are created equal: the visual systems' prior expectations may bias the system to interpret motion in terms of biologically plausible shape transformations.

Locality Preserving Refinement for Shape Matching with Functional Maps

In this paper, we address the nonrigid shape matching with outliers by a novel and effective pointwise map refinement method, termed Locality Preserving Refinement. For accurate pointwise conversion from a given functional map, our method formulates a two-step procedure. Firstly, starting with noisy point-to-point correspondences, we identify inliers by leveraging the neighborhood support, which yields a closed-form solution with linear time complexity. After obtained the reliable correspondences of inliers, we refine the pointwise correspondences for outliers using local linear embedding, which operates in an adaptive spectral similarity space to further eliminate the ambiguities that are difficult to handle in the functional space. By refining pointwise correspondences with local consistency thus embedding geometric constraints into functional spaces, our method achieves considerable improvement in accuracy with linearithmic time and space cost. Extensive experiments on public benchmarks demonstrate the superiority of our method over the state-of-the-art methods. Our code is publicly available at https://github.com/XiaYifan1999/LOPR.

Improved biharmonic kernel signature for 3D non-rigid shape matching and retrieval

Improved biharmonic kernel signature for 3D non-rigid shape matching and retrieval

Learning Robust Point Representation for 3D Non-Rigid Shape Retrieval

Learning Robust Point Representation for 3D Non-Rigid Shape Retrieval

Automated deviation-aware landmark selection for freeform product accuracy qualification in 3D printing

Landmarks are essential in non-rigid shape registration for identifying the correspondence between designs and actual products. In 3D printing, manual selection of landmarks becomes labor-intensive, due to complex product geometries and their non-uniform shape deviations. Automatic selection, however, has to pinpoint landmarks indicative of geometric regions prone to deviations for accuracy qualification. Existing automatic landmarking methods often generate clustered and redundant landmarks for prominent features with high curvatures, compromising the balance between global and local registration errors. To address these issues, we propose an automatic landmark selection method through deviation-aware segmentation and landmarking. As opposed to segmentation for semantic feature identification, deviation-aware segmentation partitions a freeform product for high-curvature region identification. Prone to deviation, these regions are generated through curvature-sensitive remeshing to extract vertices of high curvature and automatic clustering of vertices based on vertex density. Within each segment or high-curvature region, a curvature-weighted function is tailored for the Gaussian process landmarking to sequentially select landmarks with the highest local curvatures. Furthermore, we propose a new evaluation criterion to assess the effectiveness of selected landmarks through registration. The proposed approach is tested through automatic landmarking of printed dental models.

Partial Matching of Nonrigid Shapes by Learning Piecewise Smooth Functions

AbstractLearning functions defined on non‐flat domains, such as outer surfaces of non‐rigid shapes, is a central task in computer vision and geometry processing. Recent studies have explored the use of neural fields to represent functions like light reflections in volumetric domains and textures on curved surfaces by operating in the embedding space. Here, we choose a different line of thought and introduce a novel formulation of partial shape matching by learning a piecewise smooth function on a surface. Our method begins with pairing sparse landmarks defined on a full shape and its part, using feature similarity. Next, a neural representation is optimized to fit these landmarks, efficiently interpolating between the matched features that act as anchors. This process results in a function that accurately captures the partiality. Unlike previous methods, the proposed neural model of functions is intrinsically defined on the given curved surface, rather than the classical embedding Euclidean space. This representation is shown to be particularly well‐suited for representing piecewise smooth functions. We further extend the proposed framework to the more challenging part‐to‐part setting, where both shapes exhibit missing parts. Comprehensive experiments highlight that the proposed method effectively addresses partiality in shape matching and significantly outperforms leading state‐of‐the‐art methods in challenging benchmarks. Code is available at https://github.com/davidgip74/Learning-Partiality-with-Implicit-Intrinsic-Functions

RFMNet: Robust Deep Functional Maps for unsupervised non-rigid shape correspondence

In traditional deep functional maps for non-rigid shape correspondence, estimating a functional map including high-frequency information requires enough linearly independent features via the least square method, which is prone to be violated in practice, especially at an early stage of training, or costly post-processing, e.g. ZoomOut. In this paper, we propose a novel method called RFMNet (Robust Deep Functional Map Networks), which jointly considers training stability and more geometric shape features than previous works. We directly first produce a pointwise map by resorting to optimal transport and then convert it to an initial functional map. Such a mechanism mitigates the requirements for the descriptor and avoids the training instabilities resulting from the least square solver. Benefitting from the novel strategy, we successfully integrate a state-of-the-art geometric regularization for further optimizing the functional map, which substantially filters the initial functional map. We show our novel computing functional map module brings more stable training even under encoding the functional map with high-frequency information and faster convergence speed. Considering the pointwise and functional maps, an unsupervised loss is presented for penalizing the correspondence distortion of Delta functions between shapes. To catch discretization-resistant and orientation-aware shape features with our network, we utilize DiffusionNet as a feature extractor. Experimental results demonstrate our apparent superiority in correspondence quality and generalization across various shape discretizations and different datasets compared to the state-of-the-art learning methods.

Soft tissue monitoring of the surgical field: detection and tracking of breast surface deformations.

Deformable object tracking is common in the computer vision field, with applications typically focusing on nonrigid shape detection and usually not requiring specific 3D point localization.In surgical guidance however, accurate navigation is intrinsically linked to precise correspondence of tissue structure. This work presents a contactless, automated fiducial acquisition method using stereo video of the operating field to provide reliable fiducial localization for an image guidance framework in breast conserving surgery. On n=8 breasts from healthy volunteers, the breast surface was measured throughout the full range of arm motion in a supine mock-surgical position. Using hand-drawn inked fiducials, adaptive thresholding, and KAZE feature matching, precise three-dimensional fiducial locations were detected and tracked through tool interference, partial and complete marker occlusions, significant displacements and nonrigid shape distortions. Compared to digitization with a conventional optically tracked stylus, fiducials were automatically localized with 1.6 ± 0.5 mm accuracy and the two measurement methods did not significantly differ. The algorithm provided an average false discovery rate <0.1% with all cases' rates below 0.2%. On average, 85.6 ± 5.9% of visible fiducials were automatically detected and tracked, and 99.1 ± 1.1% of frames provided only true positive fiducial measurements, which indicates the algorithm achieves a data stream that can be used for reliable on-line registration. Tracking is robust to occlusions, displacements, and most shape distortions. This work-flow friendly data collection method provides highly accurate and precise three-dimensional surface data to drive an image guidance system for breast conserving surgery.

MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-robust Classifier

We offer a method for one-shot mask-guided image synthesis that allows controlling manipulations of a single image by inverting a quasi-robust classifier equipped with strong regularizers. Our proposed method, entitled MAGIC, leverages structured gradients from a pre-trained quasi-robust classifier to better preserve the input semantics while preserving its classification accuracy, thereby guaranteeing credibility in the synthesis. Unlike current methods that use complex primitives to supervise the process or use attention maps as a weak supervisory signal, MAGIC aggregates gradients over the input, driven by a guide binary mask that enforces a strong, spatial prior. MAGIC implements a series of manipulations with a single framework achieving shape and location control, intense non-rigid shape deformations, and copy/move operations in the presence of repeating objects and gives users firm control over the synthesis by requiring to simply specify binary guide masks. Our study and findings are supported by various qualitative comparisons with the state-of-the-art on the same images sampled from ImageNet and quantitative analysis using machine perception along with a user survey of 100+ participants that endorse our synthesis quality.

Extracting a functional representation from a dictionary for non-rigid shape matching

Shape matching is a fundamental problem in computer graphics with many applications. Functional maps translate the point-wise shape-matching problem into its functional counterpart and have inspired numerous solutions over the last decade. Nearly all the solutions based on functional maps rely on the eigenfunctions of the Laplace–Beltrami Operator (LB) to describe the functional spaces defined on the surfaces and then convert the functional correspondences into point-wise correspondences. However, this final step is often error-prone and inaccurate in tiny regions and protrusions, where the energy of LB does not uniformly cover the surface. We propose a new functional basis Principal Components of a Dictionary (PCD) to address such intrinsic limitation. PCD constructs an orthonormal basis from the Principal Component Analysis (PCA) of a dictionary of functions defined over the shape. These dictionaries can target specific properties of the final basis, such as achieving an even spreading of energy. Our experimental evaluation compares seven different dictionaries on established benchmarks, showing that PCD is suited to target different shape-matching scenarios, resulting in more accurate point-wise maps than the LB basis when used in the same pipeline. This evidence provides a promising alternative for improving correspondence estimation, confirming the power and flexibility of functional maps.

CA-CGNet: Component-Aware Capsule Graph Neural Network for Non-Rigid Shape Correspondence

3D non-rigid shape correspondence is significant but challenging in computer graphics, computer vision, and related fields. Although some deep neural networks have achieved encouraging results in shape correspondence, due to the complexity of the local deformation of non-rigid shapes, the ability of these networks to identify the spatial changes of objects is still insufficient. In this paper, we design a Component-aware Capsule Graph Network (CA-CGNet) to further address the features of embedding space based on the component constraints. Specifically, the dynamic clustering strategy is used to classify the features of patches produced by over-segmentation in order to further reduce noise interference. Moreover, aiming at the problem that existing routing ignores the embedding relationship between capsules, we propose a component-aware capsule graph routing to fully describe the relationship between capsules, which regards capsules as nodes in the graph network and constrains nodes through component information. Then, a knowledge distillation strategy is introduced to improve the convergence speed of the network by decreasing the parameters while maintaining accuracy. Finally, a component pair constraint is added to the functional map, and the component-based semantic loss function is proposed, which can compute isomeric in both direct and symmetric directions. The experimental results show that CA-CGNet improves by 10.21% compared with other methods, indicating the accuracy, generalization, and efficiency of our method on the FAUST, SCAPE, TOSCA, and KIDS datasets.

Multiway Non-Rigid Point Cloud Registration via Learned Functional Map Synchronization.

We present SyNoRiM, a novel way to jointly register multiple non-rigid shapes by synchronizing the maps that relate learned functions defined on the point clouds. Even though the ability to process non-rigid shapes is critical in various applications ranging from computer animation to 3D digitization, the literature still lacks a robust and flexible framework to match and align a collection of real, noisy scans observed under occlusions. Given a set of such point clouds, our method first computes the pairwise correspondences parameterized via functional maps. We simultaneously learn potentially non-orthogonal basis functions to effectively regularize the deformations, while handling the occlusions in an elegant way. To maximally benefit from the multi-way information provided by the inferred pairwise deformation fields, we synchronize the pairwise functional maps into a cycle-consistent whole thanks to our novel and principled optimization formulation. We demonstrate via extensive experiments that our method achieves a state-of-the-art performance in registration accuracy, while being flexible and efficient as we handle both non-rigid and multi-body cases in a unified framework and avoid the costly optimization over point-wise permutations by the use of basis function maps.

Automatic Identification of Aurora Fold Structure in All-Sky Images

Identification of small-scale auroral structures is key to searching for auroral events. However, it is impracticable for humans to manually select sufficient aurora events for statistical analysis, and it is also challenging for computers because of the non-rigid shape and fluid nature of auroras. Fold structure is the most common type of auroral small-scale structure, and its appearance is indicative of a variety of auroral events. This paper proposes a small-scale aurora structure identification framework to automatically detect aurora fold structures. First, the location and shape of auroras are identified based on a deep learning segmentation network. Then, the skeleton of the auroral shape is extracted to represent the trajectories of auroras. Finally, the proposed skeleton-based fold identification module (SFIM) can detect the aurora fold structure. To evaluate the effectiveness of the proposed method, we built an aurora fold structure sample dataset, namely F-dataset, containing 2000 images at 557.7 nm obtained by the all-sky imagers at Yellow River Station (YRS), Ny-Ålesund, Svalbard. Experimental results show that automatic identification can achieve good consistency with human perception. Statistical analysis of over 30,000 images shows that the fold occurrence has a distinct double-peak distribution at pre-noon and post-noon.

Learning isometry-invariant representations for point cloud analysis

3D shape analysis has drawn broad attention due to its increasing demands in various fields. Despite that impressive performance has been achieved on several databases, most researchers focus their efforts on improving the performance of shape classification, retrieval, segmentation, etc. They neglect the fact that the disturbances, such as orientation and deformation, may impact much on the perception, restricting the capacity of generalizing to real applications where the prior of orientation and pose is often unknown. In this paper, we conduct shape analysis on point clouds and propose the point projection feature, which is rotation-invariant. Specifically, a novel architecture is designed to mine features of different levels. We adopt a PointNet-based backbone to extract global feature for the point cloud, and the graph aggregation operation to perceive local pose variance in the Euclidean space or geodesic space. An efficient key point descriptor is designed to assign each point with different response and help recognize the overall geometry. Furthermore, a novel dataset, PKUnon-rigid, is built that is composed of non-rigid 3D objects, based on which we evaluate the capacity of several mainstream methods in terms of processing non-rigid shapes. Mathematical analyses and experimental results demonstrate that the proposed method can extract isometry-invariant representations for 3D shape analysis tasks without rotation augmentation, and outperforms other state-of-the-art methods. The proposed dataset is publicly available at https://github.com/tasx0823/PKUnon-rigid.

A Multi-Level Consistency Network for High-Fidelity Virtual Try-On

The 2D virtual try-on task aims to transfer a target clothing image to the corresponding region of a person image. Although an extensive amount of research has been conducted due to its immense applications, this task still remains a great challenge to handle some complicated issues (e.g., non-rigid shapes, large occlusions and arbitrary poses). To this end, we propose a novel network with structural and textural consistency-preserving mechanism for producing high-fidelity try-on images. Specifically, we first generate the semantic layout of a clothing-agnostic person to obtain the segmentation map, which is used as the transforming conditions of the target clothes. Based on a recurrent network structure, the transform lookup is performed to iteratively update a dense flow. Then, we adopt a thin-plate-spline-based warping method to estimate the coarse offset flow for all key-point positions. Guided by this sparse flow, a multi-scale deformable convolution module is designed to further iteratively predict the fine offsets for densely sampled positions, by which the clothing item and person shape can be accurately aligned. Finally, we develop a refinement module to effectively fuse the global and local features, which can render accurate geometric structures of the body parts and maintain texture sharpness of the clothes. Extensive experiments on benchmark datasets demonstrate that our method outperforms other state-of-the-art methods in terms of quantitative and qualitative try-on results. The code is available on: https://github.com/TJU-WEIHAO/MLCN .

Non-Isometric Shape Matching via Functional Maps on Landmark-Adapted Bases.

We propose a principled approach for non-isometric landmark-preserving non-rigid shape matching. Our method is based on the functional map framework, but rather than promoting isometries we focus on near-conformal maps that preserve landmarks exactly. We achieve this, first, by introducing a novel landmark-adapted basis using an intrinsic Dirichlet-Steklov eigenproblem. Second, we establish the functional decomposition of conformal maps expressed in this basis. Finally, we formulate a conformally-invariant energy that promotes high-quality landmark-preserving maps, and show how it can be optimized via a variant of the recently proposed ZoomOut method that we extend to our setting. Our method is descriptor-free, efficient and robust to significant mesh variability. We evaluate our approach on a range of benchmark datasets and demonstrate state-of-the-art performance on non-isometric benchmarks and near state-of-the-art performance on isometricones.

3DPointCaps++: Learning 3D Representations with Capsule Networks

We present 3DPointCaps++ for learning robust, flexible and generalizable 3D object representations without requiring heavy annotation efforts or supervision. Unlike conventional 3D generative models, our algorithm aims for building a structured latent space where certain factors of shape variations, such as object parts, can be disentangled into independent sub-spaces. Our novel decoder then acts on these individual latent sub-spaces (i.e. capsules) using deconvolution operators to reconstruct 3D points in a self-supervised manner. We further introduce a cluster loss ensuring that the points reconstructed by a single capsule remain local and do not spread across the object uncontrollably. These contributions allow our network to tackle the challenging tasks of part segmentation, part interpolation/replacement as well as correspondence estimation across rigid / non-rigid shape, and across / within category. Our extensive evaluations on ShapeNet objects and human scans demonstrate that our network can learn generic representations that are robust and useful in many applications.

Deep Graph Matching Based Dense Correspondence Learning Between Non-Rigid Point Clouds

Building point-to-point dense correspondence between non-rigid shapes is a fundamental and challenging problem. Although functional map-based methods which calculate basis and convert point-wise map to functional map have shown promising performance on meshes, they are not directly applicable to point clouds. Recently, learning-based frameworks were proposed to estimate correspondence directly using extracted features from non-rigid shapes, but they could not handle serious deformation well, especially with highly symmetrical and distorted shapes. In this letter, we propose a deep graph matching based end-to-end learning framework for building dense correspondence between non-rigid point clouds. Unlike functional map-based methods or recent learning-based studies, here we formulate correspondence estimation as a graph matching problem and use deep graph matching to better utilize both local geometry and global structure information. Specifically, we propose a Graph Generator, which takes high-dimensional features as nodes and calculates point-wise Euclidean distance as edges to build graphs. After the graphs’ topological structure information is embedded into nodes by the Graph Embedding Layer, the Affinity Layer and Sinkhorn Layer are used to estimate correspondence matrix between two point clouds finally. Extensive experiments on both synthetic and real scanned datasets show that the proposed method not only achieves better quantitative and qualitative performance, but also has better generalization than existing methods.