Signed Distance Field Research Articles

AiSDF: Structure-Aware Neural Signed Distance Fields in Indoor Scenes

AiSDF: Structure-Aware Neural Signed Distance Fields in Indoor Scenes

CharacterMixer: Rig‐Aware Interpolation of 3D Characters

AbstractWe present CharacterMixer, a system for blending two rigged 3D characters with different mesh and skeleton topologies while maintaining a rig throughout interpolation. CharacterMixer also enables interpolation during motion for such characters, a novel feature. Interpolation is an important shape editing operation, but prior methods have limitations when applied to rigged characters: they either ignore the rig (making interpolated characters no longer posable) or use a fixed rig and mesh topology. To handle different mesh topologies, CharacterMixer uses a signed distance field (SDF) representation of character shapes, with one SDF per bone. To handle different skeleton topologies, it computes a hierarchical correspondence between source and target character skeletons and interpolates the SDFs of corresponding bones. This correspondence also allows the creation of a single “unified skeleton” for posing and animating interpolated characters. We show that CharacterMixer produces qualitatively better interpolation results than two state‐of‐the‐art methods while preserving a rig throughout interpolation. Project page: https://seanxzhan.github.io/projects/CharacterMixer.

Real-time collision detection between general SDFs

Signed Distance Fields (SDFs) have found widespread utility in collision detection applications due to their superior query efficiency and ability to represent continuous geometries. However, little attention has been paid to calculating the intersection of two arbitrary SDFs. In this paper, we propose a novel, accurate, and real-time approach for SDF-based collision detection between two solids, both represented as SDFs. Our primary strategy entails using interval calculations and the SDF gradient to guide the search for intersection points within the geometry. For arbitrary objects, we take inspiration from existing collision detection pipelines and segment the two SDFs into multiple parts with bounding volumes. Once potential collisions between two parts are identified, our method quickly computes comprehensive intersection information such as penetration depth, contact points, and contact normals. Our method is general in that it accepts both continuous and discrete SDF representations. Experiment results show that our method can detect collisions in high-precision models in real time, highlighting its potential for a wide range of applications in computer graphics and virtual reality.

Hybrid dual mean-teacher network with double-uncertainty guidance for semi-supervised segmentation of magnetic resonance images

Semi-supervised learning has made significant progress in medical image segmentation. However, existing methods primarily utilize information from a single dimensionality, resulting in sub-optimal performance on challenging magnetic resonance imaging (MRI) data with multiple segmentation objects and anisotropic resolution. To address this issue, we present a Hybrid Dual Mean-Teacher (HD-Teacher) model with hybrid, semi-supervised, and multi-task learning to achieve effective semi-supervised segmentation. HD-Teacher employs a 2D and a 3D mean-teacher network to produce segmentation labels and signed distance fields from the hybrid information captured in both dimensionalities. This hybrid mechanism allows HD-Teacher to utilize features from 2D, 3D, or both dimensions as needed. Outputs from 2D and 3D teacher models are dynamically combined based on confidence scores, forming a single hybrid prediction with estimated uncertainty. We propose a hybrid regularization module to encourage both student models to produce results close to the uncertainty-weighted hybrid prediction to further improve their feature extraction capability. Extensive experiments of binary and multi-class segmentation conducted on three MRI datasets demonstrated that the proposed framework could (1) significantly outperform state-of-the-art semi-supervised methods (2) surpass a fully-supervised VNet trained on substantially more annotated data, and (3) perform on par with human raters on muscle and bone segmentation task. Code will be available at https://github.com/ThisGame42/Hybrid-Teacher.

QuickCSGModeling: Quick CSG Operations Based on Fusing Signed Distance Fields for VR Modeling

The latest advancements in Virtual Reality (VR) enable the creation of 3D models within a holographic immersive simulation environment. In this article, we create QuickCSGModeling , a user-friendly mid-air interactive modeling system. We first prepare a dataset consisting of diverse components and precompute the discrete signed distance function (SDF) for each component. During the modeling phase, users can freely design complicated shapes with a pair of VR controllers. Based on the discrete SDF representation, any CSG-like operation (union, intersection, and subtraction) can be performed voxel-wisely. Also, we maintain a single dynamic SDF for the whole scene, whose zero-level set surface exactly encodes the most recent constructed shape. Both SDF fusion and surface extraction are implemented via GPU for a smooth user experience. A total of 34 volunteers were asked to create their favorite models using QuickCSGModeling. With a simple training, most of them can create a fascinating shape or even a descriptive scene quickly. We also discuss how to extend our system to create articulated models with hinges, where an adaptive cube subdivision has to be enforced to improve the reconstruction accuracy around the hinge part, followed by a Dual Contouring-based surface extraction. 1

LISR: Learning Linear 3D Implicit Surface Representation Using Compactly Supported Radial Basis Functions

Implicit 3D surface reconstruction of an object from its partial and noisy 3D point cloud scan is the classical geometry processing and 3D computer vision problem. In the literature, various 3D shape representations have been developed, differing in memory efficiency and shape retrieval effectiveness, such as volumetric, parametric, and implicit surfaces. Radial basis functions provide memory-efficient parameterization of the implicit surface. However, we show that training a neural network using the mean squared error between the ground-truth implicit surface and the linear basis-based implicit surfaces does not converge to the global solution. In this work, we propose locally supported compact radial basis functions for a linear representation of the implicit surface. This representation enables us to generate 3D shapes with arbitrary topologies at any resolution due to their continuous nature. We then propose a neural network architecture for learning the linear implicit shape representation of the 3D surface of an object. We learn linear implicit shapes within a supervised learning framework using ground truth Signed-Distance Field (SDF) data for guidance. The classical strategies face difficulties in finding linear implicit shapes from a given 3D point cloud due to numerical issues (requires solving inverse of a large matrix) in basis and query point selection. The proposed approach achieves better Chamfer distance and comparable F-score than the state-of-the-art approach on the benchmark dataset. We also show the effectiveness of the proposed approach by using it for the 3D shape completion task.

O^2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model

Occlusion is a common issue in 3D reconstruction from RGB-D videos, often blocking the complete reconstruction of objects and presenting an ongoing problem. In this paper, we propose a novel framework, empowered by a 2D diffusion-based in-painting model, to reconstruct complete surfaces for the hidden parts of objects. Specifically, we utilize a pre-trained diffusion model to fill in the hidden areas of 2D images. Then we use these in-painted images to optimize a neural implicit surface representation for each instance for 3D reconstruction. Since creating the in-painting masks needed for this process is tricky, we adopt a human-in-the-loop strategy that involves very little human engagement to generate high-quality masks. Moreover, some parts of objects can be totally hidden because the videos are usually shot from limited perspectives. To ensure recovering these invisible areas, we develop a cascaded network architecture for predicting signed distance field, making use of different frequency bands of positional encoding and maintaining overall smoothness. Besides the commonly used rendering loss, Eikonal loss, and silhouette loss, we adopt a CLIP-based semantic consistency loss to guide the surface from unseen camera angles. Experiments on ScanNet scenes show that our proposed framework achieves state-of-the-art accuracy and completeness in object-level reconstruction from scene-level RGB-D videos. Code: https://github.com/THU-LYJ-Lab/O2-Recon.

Multi-scale hash encoding based neural geometry representation

Recently, neural implicit function-based representation has attracted more and more attention, and has been widely used to represent surfaces using differentiable neural networks. However, surface reconstruction from point clouds or multi-view images using existing neural geometry representations still suffer from slow computation and poor accuracy. To alleviate these issues, we propose a multi-scale hash encoding-based neural geometry representation which effectively and efficiently represents the surface as a signed distance field. Our novel neural network structure carefully combines low-frequency Fourier position encoding with multi-scale hash encoding. The initialization of the geometry network and geometry features of the rendering module are accordingly redesigned. Our experiments demonstrate that the proposed representation is at least 10 times faster for reconstructing point clouds with millions of points. It also significantly improves speed and accuracy of multi-view reconstruction. Our code and models are available at https://github.com/Dengzhi-USTC/Neural-Geometry-Reconstruction.

NeuralVDB: High-resolution Sparse Volume Representation using Hierarchical Neural Networks

We introduce NeuralVDB, which improves on an existing industry standard for efficient storage of sparse volumetric data, denoted VDB [Museth 2013 ], by leveraging recent advancements in machine learning. Our novel hybrid data structure can reduce the memory footprints of VDB volumes by orders of magnitude, while maintaining its flexibility and only incurring small (user-controlled) compression errors. Specifically, NeuralVDB replaces the lower nodes of a shallow and wide VDB tree structure with multiple hierarchical neural networks that separately encode topology and value information by means of neural classifiers and regressors respectively. This approach is proven to maximize the compression ratio while maintaining the spatial adaptivity offered by the higher-level VDB data structure. For sparse signed distance fields and density volumes, we have observed compression ratios on the order of 10× to more than 100× from already compressed VDB inputs, with little to no visual artifacts. Furthermore, NeuralVDB is shown to offer more effective compression performance compared to other neural representations such as Neural Geometric Level of Detail [Takikawa et al. 2021 ], Variable Bitrate Neural Fields [Takikawa et al. 2022a ], and Instant Neural Graphics Primitives [Müller et al. 2022 ]. Finally, we demonstrate how warm-starting from previous frames can accelerate training, i.e., compression, of animated volumes as well as improve temporal coherency of model inference, i.e., decompression.

A practical algorithm for the design of multiple-sized porous scaffolds with triply periodic structures

In this study, we present a practical volume-merging method for generating multiple-sized porous structures that exhibit geometries with triply periodic minimal surface (TPMS) lattice structures. The proposed method consists of three stages: (1) designing the physical models with a signed distance field, (2) performing a merging operation for the porous scaffolds, and (3) assembling different units into a composite structure. The significant advantages of the proposed algorithm can be summarized as follows: Our method is independent of the model shape; the designed structures maintain a smooth surface with a constant mean curvature, and the mathematical computational complexity is low. We can join two different-sized triply periodic minimal surface lattices in the radial direction, where the transition region is obtained by smooth interpolation between the two lattice structures with different cell sizes or types. However, constructing large-sized models is only conceptually possible due to computational cost and memory storage constraints. To overcome these limitations, we present a practical method that can efficiently assemble large-scaled models at a low computational cost. The proposed method is based on a Boolean union operation of basic units of TPMS. Thus, it is simple to generate large-scale three-dimensional multiple-sized porous volumes based on our proposed method, which can be applied to many applications in mechanical and electrical engineering. The produced multi-scale compound scaffolds have smooth surfaces without fractures, making them suitable for straightforward application in additive manufacturing. Several numerical tests are conducted to validate the efficiency of the proposed algorithm.

A general integration kernel formulation for immersed boundary method

This study proposes a new robust and accurate immersed boundary method for the immersion of solid bodies within a fluid with a Cartesian grid. The present method introduces the signed distance fields to recognize the immersed geometry contours, eliminating the need of Lagrangian points. To fully maximize the advantages offered by signed distance fields, a general integration kernel formulation is introduced into the direct forcing method to replace the conventional regularized delta function. With the combination of signed distance fields and kernel function, an interpolation along the radial direction instead of three-dimensional directions is feasible, which further reduces the extra calculation cost involved by immersed boundary method. The numerical results at low Reynolds numbers are compared to experimental and previous numerical results, which shows the efficiency and accuracy of this new method. Upon thorough validation, the proposed method in this paper demonstrates excellent performance across various scenarios, including static and moving cases as well as two- and three-dimensional configurations. And our method greatly reduces the cost of pretreatment of immersed geometry contours and apparently improves the convenience of the method.

DPO-SDF: Differentiable Physical Optics Method With Signed Distance Field for 3-D Reconstruction of Perfect Electric Conductors

DPO-SDF: Differentiable Physical Optics Method With Signed Distance Field for 3-D Reconstruction of Perfect Electric Conductors

Learning Implicit Fields for Point Cloud Filtering.

Since point clouds acquired by scanners inevitably contain noise, recovering a clean version from a noisy point cloud is essential for further 3D geometry processing applications. Several data-driven approaches have been recently introduced to overcome the drawbacks of traditional filtering algorithms, such as less robust preservation of sharp features and tedious tuning for multiple parameters. Most of these methods achieve filtering by directly regressing the position/displacement of each point, which may blur detailed features and is prone to uneven distribution. In this paper, we propose a novel data-driven method that explores the implicit fields. Our assumption is that the given noisy points implicitly define a surface, and we attempt to obtain a point's movement direction and distance separately based on the predicted signed distance fields (SDFs). Taking a noisy point cloud as input, we first obtain a consistent alignment by incorporating the global points into local patches. We then feed them into an encoder-decoder structure and predict a 7D vector consisting of SDFs. Subsequently, the distance can be obtained directly from the first element in the vector, and the movement direction can be obtained by computing the gradient descent from the last six elements (i.e., six surrounding SDFs). We finally obtain the filtered results by moving each point with its predicted distance along its movement direction. Our method can produce feature-preserving results without requiring explicit normals. Experiments demonstrate that our method visually outperforms state-of-the-art methods and generally produces better quantitative results than position-based methods (both learning and non-learning).

Animatable Implicit Neural Representations for Creating Realistic Avatars From Videos.

This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images. However, they represent the deformation field as translational vector field or SE(3) field, which makes the optimization highly under-constrained. Moreover, these representations cannot be explicitly controlled by input motions. Instead, we introduce blend weight fields to produce the deformation fields. Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-to-canonical and canonical-to-observation correspondences. Since 3D human skeletons are more observable, they can regularize the learning of deformation fields. Moreover, the blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model. To improve the quality of human modeling, we further represent the human geometry as a signed distance field in the canonical space. Additionally, a neural point displacement field is introduced to enhance the capability of the blend weight field on modeling detailed human motions. Experiments show that our approach significantly outperforms recent human modeling methods.

HSDF: Hybrid Sign and Distance Field for Neural Representation of Surfaces with Arbitrary Topologies.

Neural implicit function based on signed distance field (SDF) has achieved impressive progress in reconstructing 3D models with high fidelity. However, such approaches can only represent closed surfaces. Recent works based on unsigned distance function (UDF) are proposed to handle both watertight and single-layered open surfaces. Nonetheless, as UDF is signless, its direct output is limited to the point cloud, which imposes an additional challenge on extracting high-quality meshes from discrete points. To address this challenge, we present a novel neural implicit representation coded HSDF, which is a hybrid of signed and unsigned distance fields. In particular, HSDF is able to represent arbitrary topologies containing both closed and open surfaces while being compatible with existing iso-surface extraction techniques for easy field-to-mesh conversion. In addition to predicting a UDF, we propose to learn an additional sign field. Unlike traditional SDF, HSDF is able to locate the surface of interest before level surface extraction by generating surface points following NDF[1]. We are then able to obtain open surfaces via an adaptive meshing approach that only instantiates regions containing surfaces into a polygon mesh. HSDF benefits downstream tasks like neural rendering, as it enables the rendering of back-faces of open surfaces. We also propose HSDF-Net, a dedicated learning framework that factorizes the learning of HSDF into two easier sub-problems. Experiments and evaluations show that HSDF outperforms the state-of-the-art techniques both qualitatively and quantitatively on some of the used datasets.

A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields.

We present the first real-time method for inserting a rigid virtual object into a neural radiance field (NeRF), which produces realistic lighting and shadowing effects, as well as allows interactive manipulation of the object. By exploiting the rich information about lighting and geometry in a NeRF, our method overcomes several challenges of object insertion in augmented reality. For lighting estimation, we produce accurate and robust incident lighting that combines the 3D spatially-varying lighting from NeRF and an environment lighting to account for sources not covered by the NeRF. For occlusion, we blend the rendered virtual object with the background scene using an opacity map integrated from the NeRF. For shadows, with a precomputed field of spherical signed distance fields, we query the visibility term for any point around the virtual object, and cast soft, detailed shadows onto 3D surfaces. Compared with state-of-the-art techniques, our approach can insert virtual objects into scenes with superior fidelity, and has great potential to be further applied to augmented reality systems.

Range Map Interpolation-Based 3-D LiDAR Truncated Signed Distance Fields Mapping in Outdoor Environments

Range Map Interpolation-Based 3-D LiDAR Truncated Signed Distance Fields Mapping in Outdoor Environments

Localization Exploiting Semantic and Metric Information in Non-static Indoor Environments

Mobile robot localization is an important task in navigation and can be challenging, especially in non-static environments as the scene naturally involves movable objects and appearance changes. In this paper, we address the problem of estimating the robot’s pose in non-static environments containing movable objects. We understand as non-static environments, dynamic environments in which objects might be moved or changed their appearance. We propose a probabilistic localization approach that combines metric and semantic information and takes into account both, static and movable objects. We perform a pixel-wise association of depth and semantic data from an RGB-D sensor with a semantically-augmented truncated signed distance field (TSDF) in order to estimate the robot’s pose. The combination of metric and semantic information increases the robustness w.r.t. movable objects and object appearance changes. The experiments conducted in a real indoor environment and a publicly-available dataset suggest that our approach successfully estimates robot pose in non-static environments and they show an improvement compared to robot localization based only on metric or semantic information and compared to a feature-based method.

City-scale continual neural semantic mapping with three-layer sampling and panoptic representation

Neural implicit representations are drawing much attention from the robotics community recently, as they are expressive, continuous and compact. However, city-scale continual implicit dense mapping based on sparse LiDAR input is still an under-explored challenge. To this end, we successfully build a city-scale continual neural mapping system with a panoptic representation that consists of environment-level and instance-level modeling. Given a stream of sparse LiDAR point cloud, it maintains a model that maps 3D coordinates to signed distance field (SDF) values. To address the difficulty of representing geometric information at different levels in city-scale space, we propose a tailored three-layer sampling strategy to sample the global, local and near-surface domains dynamically. Meanwhile, to realize high fidelity mapping of instance under incomplete observation, category-specific prior is introduced to better model the geometric details. We evaluate on the public SemanticKITTI [1] dataset and demonstrate the significance of the newly proposed three-layer sampling strategy and panoptic representation, using both quantitative and qualitative results. Codes and model will be publicly available.

IF-TONIR: Iteration-free Topology Optimization based on Implicit Neural Representations

Topology optimization holds great significance as a research topic in the field of mechanical engineering, aiming to design and optimize structures to achieve desired performance while adhering to specific constraints. However, its high computational complexity and iterative optimization process severely impact the efficiency, which presents substantial obstacles to its practical applications. To tackle this challenge, recent research is dedicated to the advancement of iteration-free topology optimization methods that leverage neural networks and deep learning, aiming to directly predict optimal structures through optimization problem configurations. In this paper, we propose IF-TONIR, a novel data-driven topology optimization method that utilizes implicit neural representations. Our approach employs signed distance fields to represent structures, offering compact and smooth representations that effectively eliminate the checkerboard phenomenon commonly observed in density-based methods. IF-TONIR leverages Conditional Variational Autoencoders, which use a CNN-based encoder and a MLP-based decoder to learn and reconstruct optimal structures. We employ the features extracted from physical information as conditions to guide the decoder in generating optimal structures that adhere to specific design domain shapes and boundary conditions. Furthermore, we propose the integration of a topological loss based on persistent homology to train the model. This loss function effectively penalizes the existence of structural disconnections in the reconstructed output, thereby enhancing the overall physical reliability of the generated structures. Various experiments have demonstrated that our iteration-free topology optimization method based on implicit representations can accurately identify regions of high strain energy and generate continuous structures with low compliance. The methods also holds the theoretical capability of outputting optimal structures at any desired resolution. Our code and dataset are available on https://github.com/jbHu67/IF-TONIR.git