Dense Correspondence Research Articles

SPDepth: Enhancing Self-Supervised Indoor Monocular Depth Estimation via Self-Propagation

Due to the existence of low-textured areas in indoor scenes, some self-supervised depth estimation methods have specifically designed sparse photometric consistency losses and geometry-based losses. However, some of the loss terms cannot supervise all the pixels, which limits the performance of these methods. Some approaches introduce an additional optical flow network to provide dense correspondences supervision, but overload the loss function. In this paper, we propose to perform depth self-propagation based on feature self-similarities, where high-accuracy depths are propagated from supervised pixels to unsupervised ones. The enhanced self-supervised indoor monocular depth estimation network is called SPDepth. Since depth self-similarities are significant in a local range, a local window self-attention module is embedded at the end of the network to propagate depths in a window. The depth of a pixel is weighted using the feature correlation scores with other pixels in the same window. The effectiveness of self-propagation mechanism is demonstrated in the experiments on the NYU Depth V2 dataset. The root-mean-squared error of SPDepth is 0.585 and the δ1 accuracy is 77.6%. Zero-shot generalization studies are also conducted on the 7-Scenes dataset and provide a more comprehensive analysis about the application characteristics of SPDepth.

The impact of breastfeeding on facial appearance in adolescent children.

Evidence that breastfeeding impacts the facial features of children is conflicting. Most studies to date have focused on dental and skeletal malocclusion. It currently remains unclear whether such effects are of sufficient magnitude to be detectable on outward facial appearance. Here, we evaluate the extent to which maternally reported breastfeeding is associated with 3D facial shape in a large adolescent cohort. After extracting 3D facial surfaces from MR scans in 2275 9- and 10-year-old children and aligning the surfaces in dense correspondence, we analyzed the effect of breastfeeding on shape as a dichotomous (no/yes) and semi-quantitative (to assess duration in months) variable using partial least squares regression. Our results showed no effect (p = 0.532) when breastfeeding was dichotomized. However, when treated as a semi-quantitative variable, breastfeeding duration was associated with statistically significant changes in shape (p = 3.61x 10-4). The most prominent facial changes included relative retrusion of the central midface, zygomatic arches, and orbital regions along with relative protrusion of forehead, cheek, and mandible. The net effect was that as breastfeeding duration increased, the facial profile in children became flatter (less convex). The observed effects on the face, however, were subtle and likely not conspicuous enough to be noticed by most observers. This was true even when comparing the faces of children breastfed for 19-24 months to children with no reported breastfeeding. Thus, breastfeeding does appear to have detectable effect on outward facial appearance in adolescent children, but its practical impact appears to be minimal.

Refined Prior Guided Category-Level 6D Pose Estimation and Its Application on Robotic Grasping

Estimating the 6D pose and size of objects is crucial in the task of visual grasping for robotic arms. Most current algorithms still require the 3D CAD model of the target object to match with the detected points, and they are unable to predict the object’s size, which significantly limits the generalizability of these methods. In this paper, we introduce category priors and extract high-dimensional abstract features from both the observed point cloud and the prior to predict the deformation matrix of the reconstructed point cloud and the dense correspondence between the reconstructed and observed point clouds. Furthermore, we propose a staged geometric correction and dense correspondence refinement mechanism to enhance the accuracy of regression. In addition, a novel lightweight attention module is introduced to further integrate the extracted features and identify potential correlations between the observed point cloud and the category prior. Ultimately, the object’s translation, rotation, and size are obtained by mapping the reconstructed point cloud to a normalized canonical coordinate system. Through extensive experiments, we demonstrate that our algorithm outperforms existing methods in terms of performance and accuracy on commonly used benchmarks for this type of problem. Additionally, we implement the algorithm in robotic arm-grasping simulations, further validating its effectiveness.

Prior-information-guided corresponding point regression network for 6D pose estimation

A key step in bin-picking is to obtain the 6DoF pose of the object. However, One of the reasons why many pose estimation methods fail to deal with occlusions and noise is that they lack reliable correspondence and effective utilization of CAD model information. In order to solve such issue, we propose a new prior-information-guided 6D pose estimation network based on 3D-3D correspondence prediction. Specifically, the network establishes dense correspondences between the object and the CAD model by predicting the object’s corresponding points on the CAD point cloud. This correspondence prediction method allows the network to learn more geometric details and perform better than direct regression. Furthermore, we implement the embedding of CAD model features in the network, which utilizes the prior information of the CAD model to compensate for the missing object information due to occlusion and viewpoint and provide more precise point regression. In addition, thanks to this points regression method, we can intuitively eliminate erroneous points via distance threshold judgment, thus achieving robustness to noise and outliers. Finally, the pose are then estimated through a SVD solver. Experiments show that our method outperforms most methods on LM, LM-O and YCB-V datasets, and its performance on LM-O datasets is 9% higher than the most advanced method. This proves that our method has excellent robustness and achieves the state-of-the-art performance.

Optimal Transport Approaches for Cloud Fusion in 3D Reconstruction

Cloud fusion plays a crucial role in achieving accurate 3D models. This paper introduces two optimal transport-based approaches, Gaussian Fusion and Cyclical Fusion, to showcase the effectiveness of optimal transport in establishing dense correspondence among incoming 3D point clouds. These approaches encompass several important components. Firstly, the fusion correspondences are derived from the optimal transport problem. The core principle of Gaussian Fusion and Cyclical Fusion involves manipulating points along geodesic curves to effectively leverage significant local geometric information. Secondly, this paper explores the fundamental concepts underlying Gaussian Fusion (displacement interpolation) and Cyclical Fusion (cyclical monotonicity), which greatly enhance the accuracy and completeness of the reconstruction process. And provides evidence for the uniqueness of displacement interpolation as geodesics on L2-Wasserstein space and the rationality of cyclical monotonicity in this context. Finally, the proposed approaches excel in capturing intricate surface details, particularly on small objects, when compared to the original fusion scheme which often introduces severe artifacts.

ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular Region

AbstractEye gaze and expressions are crucial non‐verbal signals in face‐to‐face communication. Visual effects and telepresence demand significant improvements in personalized tracking, animation, and synthesis of the eye region to achieve true immersion. Morphable face models, in combination with coordinate‐based neural volumetric representations, show promise in solving the difficult problem of reconstructing intricate geometry (eyelashes) and synthesizing photorealistic appearance variations (wrinkles and specularities) of eye performances. We propose a novel hybrid representation ‐ ShellNeRF ‐ that builds a discretized volume around a 3DMM face mesh using concentric surfaces to model the deformable ‘periocular’ region. We define a canonical space using the UV layout of the shells that constrains the space of dense correspondence search. Combined with an explicit eyeball mesh for modeling corneal light‐transport, our model allows for animatable photorealistic 3D synthesis of the whole eye region. Using multi‐view video input, we demonstrate significant improvements over state‐of‐the‐art in expression re‐enactment and transfer for high‐resolution close‐up views of the eye region.

Graph-Based Contrastive Learning for Description and Detection of Local Features.

Confronted with the task environment full of repetitive textures, the state-of-art description and detection methods for local features greatly suffer from the "pseudo-negatives," bringing inconsistent optimization objectives during training. To address this problem, this article develops a self-supervised graph-based contrastive learning framework to train the model for local features, GCLFeat. The proposed approach learns to alleviate the pseudo-negatives specifically from three aspects: 1) designing a graph neural network (GNN), which focuses on mining the local transformational invariance across different views and global textual knowledge within individual images; 2) generating the dense correspondence annotations from a diverse natural dataset with a self-supervised paradigm; and 3) adopting a keypoints-aware sampling strategy to compute the loss across the whole dataset. The experimental results show that the unsupervised framework outperforms the state-of-the-art supervised baselines on diverse downstream benchmarks including image matching, 3-D reconstruction and visual localization. The code will be made public and available at https://github.com/RealZihaoWang/GCLFeat.

Learning Dense Correspondence for NeRF-Based Face Reenactment

Face reenactment is challenging due to the need to establish dense correspondence between various face representations for motion transfer. Recent studies have utilized Neural Radiance Field (NeRF) as fundamental representation, which further enhanced the performance of multi-view face reenactment in photo-realism and 3D consistency. However, establishing dense correspondence between different face NeRFs is non-trivial, because implicit representations lack ground-truth correspondence annotations like mesh-based 3D parametric models (e.g., 3DMM) with index-aligned vertexes. Although aligning 3DMM space with NeRF-based face representations can realize motion control, it is sub-optimal for their limited face-only modeling and low identity fidelity. Therefore, we are inspired to ask: Can we learn the dense correspondence between different NeRF-based face representations without a 3D parametric model prior? To address this challenge, we propose a novel framework, which adopts tri-planes as fundamental NeRF representation and decomposes face tri-planes into three components: canonical tri-planes, identity deformations, and motion. In terms of motion control, our key contribution is proposing a Plane Dictionary (PlaneDict) module, which efficiently maps the motion conditions to a linear weighted addition of learnable orthogonal plane bases. To the best of our knowledge, our framework is the first method that achieves one-shot multi-view face reenactment without a 3D parametric model prior. Extensive experiments demonstrate that we produce better results in fine-grained motion control and identity preservation than previous methods.

Spotting the Unseen: Reciprocal Consensus Network Guided by Visual Archetypes

Humans often require only a few visual archetypes to spot novel objects. Based on this observation, we present a strategy rooted in ``spotting the unseen" by establishing dense correspondences between potential query image regions and a visual archetype, and we propose the Consensus Network (CoNet). Our method leverages relational patterns intra and inter images via Auto-Correlation Representation (ACR) and Mutual-Correlation Representation (MCR). Within each image, the ACR module is capable of encoding both local self-similarity and global context simultaneously. Between the query and support images, the MCR module computes the cross-correlation across two image representations and introduces a reciprocal consistency constraint, which can incorporate to exclude outliers and enhance model robustness. To overcome the challenges of low-resource training data, particularly in one-shot learning scenarios, we incorporate an adaptive margin strategy to better handle diverse instances. The experimental results indicate the effectiveness of the proposed method across diverse domains such as object detection in natural scenes, and text spotting in both historical manuscripts and natural scenes, which demonstrates its sparkling generalization ability. Our code is available at: https://github.com/infinite-hwb/conet.

Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objects.

The objective of this paper is to learn dense 3D shape correspondence for topology-varying generic objects in an unsupervised manner. Conventional implicit functions estimate the occupancy of a 3D point given a shape latent code. Instead, our novel implicit function produces a probabilistic embedding to represent each 3D point in a part embedding space. Assuming the corresponding points are similar in the embedding space, we implement dense correspondence through an inverse function mapping from the part embedding vector to a corresponded 3D point. Both functions are jointly learned with several effective and uncertainty-aware loss functions to realize our assumption, together with the encoder generating the shape latent code. During inference, if a user selects an arbitrary point on the source shape, our algorithm can automatically generate a confidence score indicating whether there is a correspondence on the target shape, as well as the corresponding semantic point if there is one. Such a mechanism inherently benefits man-made objects with different part constitutions. The effectiveness of our approach is demonstrated through unsupervised 3D semantic correspondence and shape segmentation.

Ultra-dense Motion Capture: An exploratory full-automatic approach for dense tracking of breast motion in 4D.

Understanding the dynamic deformation pattern and biomechanical properties of breasts is crucial in various fields, including designing ergonomic bras and customized prostheses, as well as in clinical practice. Previous studies have recorded and analyzed the dynamic behaviors of the breast surface using 4D scanning, which provides a sequence of 3D meshes during movement with high spatial and temporal resolutions. However, these studies are limited by the lack of robust and automated data processing methods which result in limited data coverage or error-prone analysis results. To address this issue, we identify revealing inter-frame dense correspondence as the core challenge towards conducting reliable and consistent analysis of the 4D scanning data. We proposed a fully-automatic approach named Ulta-dense Motion Capture (UdMC) using Thin-plate Spline (TPS) to augment the sparse landmarks recorded via motion capture (MoCap) as initial dense correspondence and then rectified it with a sophisticated post-alignment scheme. Two downstream tasks are demonstrated to validate its applicability: virtual landmark tracking and deformation intensity analysis. For evaluation, a dynamic 4D human breast anthropometric dataset DynaBreastLite was constructed. The results show that our approach can robustly capture the dynamic deformation characteristics of the breast surfaces, significantly outperforms baselines adapted from previous works in terms of accuracy, consistency, and efficiency. For 10 fps dataset, average error of 0.25 cm on control-landmarks and 0.33 cm on non-control (arbitrary) landmarks were achieved, with 17-70 times faster computation time. Evaluation was also carried out on 60 fps and 120 fps datasets, with consistent and large performance gaining being observed. The proposed method may contribute to advancing research in breast anthropometry, biomechanics, and ergonomics by enabling more accurate tracking of the breast surface deformation patterns and dynamic characteristics.

SCorP: Statistics-Informed Dense Correspondence Prediction Directly from Unsegmented Medical Images.

Statistical shape modeling (SSM) is a powerful computational framework for quantifying and analyzing the geometric variability of anatomical structures, facilitating advancements in medical research, diagnostics, and treatment planning. Traditional methods for shape modeling from imaging data demand significant manual and computational resources. Additionally, these methods necessitate repeating the entire modeling pipeline to derive shape descriptors (e.g., surface-based point correspondences) for new data. While deep learning approaches have shown promise in streamlining the construction of SSMs on new data, they still rely on traditional techniques to supervise the training of the deep networks. Moreover, the predominant linearity assumption of traditional approaches restricts their efficacy, a limitation also inherited by deep learning models trained using optimized/established correspondences. Consequently, representing complex anatomies becomes challenging. To address these limitations, we introduce SCorP, a novel framework capable of predicting surface-based correspondences directly from unsegmented images. By leveraging the shape prior learned directly from surface meshes in an unsupervised manner, the proposed model eliminates the need for an optimized shape model for training supervision. The strong shape prior acts as a teacher and regularizes the feature learning of the student network to guide it in learning image-based features that are predictive of surface correspondences. The proposed model streamlines the training and inference phases by removing the supervision for the correspondence prediction task while alleviating the linearity assumption. Experiments on the LGE MRI left atrium dataset and Abdomen CT-1K liver datasets demonstrate that the proposed technique enhances the accuracy and robustness of image-driven SSM, providing a compelling alternative to current fully supervised methods.

Learning Canonical Embeddings for Unsupervised Shape Correspondence With Locally Linear Transformations.

We present a new approach to unsupervised shape correspondence learning between pairs of point clouds. We make the first attempt to adapt the classical locally linear embedding algorithm (LLE)-originally designed for nonlinear dimensionality reduction-for shape correspondence. The key idea is to find dense correspondences between shapes by first obtaining high-dimensional neighborhood-preserving embeddings of low-dimensional point clouds and subsequently aligning the source and target embeddings using locally linear transformations. We demonstrate that learning the embedding using a new LLE-inspired point cloud reconstruction objective results in accurate shape correspondences. More specifically, the approach comprises an end-to-end learnable framework of extracting high-dimensional neighborhood-preserving embeddings, estimating locally linear transformations in the embedding space, and reconstructing shapes via divergence measure-based alignment of probability density functions built over reconstructed and target shapes. Our approach enforces embeddings of shapes in correspondence to lie in the same universal/canonical embedding space, which eventually helps regularize the learning process and leads to a simple nearest neighbors approach between shape embeddings for finding reliable correspondences. Comprehensive experiments show that the new method makes noticeable improvements over state-of-the-art approaches on standard shape correspondence benchmark datasets covering both human and nonhuman shapes.

PA-Pose: Partial point cloud fusion based on reliable alignment for 6D pose tracking

Learning-based 6-DOF (6D) pose tracking, serving as a basis for most real-time applications such as augmented reality and robot manipulation, receives attention transiting from 2D to 3D vision, with the popularity of depth sensors. However, the irregular nature of 3D point clouds challenges this task, especially since the lack of explicit alignments hinders the interaction and fusion between the observed point clouds. Therefore, this paper proposes a novel approach named PA-Pose to achieve 6D pose tracking in point clouds. It takes the forward-predicted dense correspondences within an overlap as reliable alignments, to guide the feature fusion of the partial-to-partial point clouds. Then, the relative transformation pose of adjacent observations is continuously regressed from the point-wisely fused features by confidence scoring, avoiding non-differentiable pose fitting. In addition, a shifted point convolution (SPConv) operation is introduced in the fusion process, to further promote the local context interaction of the observed point cloud pair in the expanded alignment field. Extensive experiments on two benchmark datasets (YCB-Video and YCBInEOAT) demonstrate that our method achieves state-of-the-art performance. Even though only 3D point clouds are taken as input, our PA-Pose is still competitive with those methods fully utilizing RGB-D information in the single view. Finally, experiments in the real scene for tracking industrial objects also validates the effectiveness of the proposed method.

GMIW-Pose: Camera Pose Estimation via Global Matching and Iterative Weighted Eight-Point Algorithm

We propose a novel approach, GMIW-Pose, to estimate the relative camera poses between two views. This method leverages a Transformer-based global matching module to obtain robust 2D–2D dense correspondences, followed by iterative refinement of matching weights using ConvGRU. Ultimately, the camera’s relative pose is determined through the weighted eight-point algorithm. Compared with the previous best two-view pose estimation method, GMIW-Pose reduced the Absolute Trajectory Error (ATE) by 24% on the TartanAir dataset; it achieved the best or second-best performance in multiple scenarios of the TUM-RGBD and KITTI datasets without fine-tuning, among which ATE decreased by 22% on the TUM-RGBD dataset.

Unifying Flow, Stereo and Depth Estimation.

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

Linear Time Non-Local Cost Aggregation on Complementary Spatial Tree Structures

Studies on many dense correspondence tasks in the field of computer vision attempt to find spatially smooth results. A typical way to solve these problems is by smoothing the matching costs using edge-preserving filters. However, local filters generate locally optimal results, in that they only take the costs over a small support window into account, and non-local filters based on a minimum spanning tree (MST) tend to overuse the piece-wise constant assumption. In this paper, we propose a linear time non-local cost aggregation method based on two complementary spatial tree structures. The geodesic distances in both the spatial and intensity spaces along the tree structures are used to evaluate the similarity of pixels, and the final aggregated cost is the sum of the outputs from these two trees. The filtering output of a pixel on each tree can be obtained by recursively aggregating the costs along eight sub-trees with linear time complexity. The only difference between the filtering procedures on these two spatial tree structures is the order of the filtering. Experimental results in optical flow estimation and stereo matching on the Middlebury and KITTI datasets demonstrate the effectiveness and efficiency of our method. It turns out that our method outperforms typical non-local filters based on the MST in cost aggregation. Moreover, a comparison of handcrafted features and deep features learned by convolutional neural networks (CNNs) in calculating the matching cost is also provided. The code will be available soon.

Groupwise Shape Correspondence Refinement with a Region of Interest Focus

AbstractWhile collections of scan shapes are becoming more prevalent in many real‐world applications, finding accurate and dense correspondences across multiple shapes remains a challenging task. In this work, we introduce a new approach for refining non‐rigid correspondences among a collection of 3D shapes undergoing non‐rigid deformation. Our approach incorporates a Region Of Interest (ROI) into the refinement process, which is specified by the user on one shape within the collection. Based on the functional map framework and more specifically on the notion of cycle‐consistency, our formulation improves the overall matching consistency while prioritizing that of the region of interest. Specifically, the initial pairwise correspondences are refined by first defining the localized harmonics that are confined within the transferred ROI on each shape, and subsequently applying the CCLB (Canonical Consistent Latent Basis) framework both on the global and the localized harmonics. This leads to an enhanced matching accuracy for both the ROIs and the overall shapes across the collection. We evaluate our method on various synthetic and real scan datasets, in comparison with the state‐of‐the‐art techniques.

MacJNet: weakly-supervised multimodal image deformable registration using joint learning framework and multi-sampling cascaded MIND

Deformable multimodal image registration plays a key role in medical image analysis. It remains a challenge to find accurate dense correspondences between multimodal images due to the significant intensity distortion and the large deformation. macJNet is proposed to align the multimodal medical images, which is a weakly-supervised multimodal image deformable registration method using a joint learning framework and multi-sampling cascaded modality independent neighborhood descriptor (macMIND). The joint learning framework consists of a multimodal image registration network and two segmentation networks. The proposed macMIND is a modality-independent image structure descriptor to provide dense correspondence for registration, which incorporates multi-orientation and multi-scale sampling patterns to build self-similarity context. It greatly enhances the representation ability of cross-modal features in the registration network. The semi-supervised segmentation networks generate anatomical labels to provide semantics correspondence for registration, and the registration network helps to improve the performance of multimodal image segmentation by providing the consistency of anatomical labels. 3D CT-MR liver image dataset with 118 samples is built for evaluation, and comprehensive experiments have been conducted to demonstrate that macJNet achieves superior performance over state-of-the-art multi-modality medical image registration methods.

Sparse-to-Dense Matching Network for Large-scale LiDAR Point Cloud Registration.

Point cloud registration is a fundamental problem in 3D computer vision. Previous learning-based methods for LiDAR point cloud registration can be categorized into two schemes: dense-to-dense matching methods and sparse-to-sparse matching methods. However, for large-scale outdoor LiDAR point clouds, solving dense point correspondences is time-consuming, whereas sparse keypoint matching easily suffers from keypoint detection error. In this paper, we propose SDMNet, a novel Sparse-to-Dense Matching Network for large-scale outdoor LiDAR point cloud registration. Specifically, SDMNet performs registration in two sequential stages: sparse matching stage and local-dense matching stage. In the sparse matching stage, we sample a set of sparse points from the source point cloud and then match them to the dense target point cloud using a spatial consistency enhanced soft matching network and a robust outlier rejection module. Furthermore, a novel neighborhood matching module is developed to incorporate local neighborhood consensus, significantly improving performance. The local-dense matching stage is followed for fine-grained performance, where dense correspondences are efficiently obtained by performing point matching in local spatial neighborhoods of high-confidence sparse correspondences. Extensive experiments on three large-scale outdoor LiDAR point cloud datasets demonstrate that the proposed SDMNet achieves state-of-the-art performance with high efficiency.