Implicit Surface Representation Research Articles

Automatic segmentation and implicit surface representation of dynamic cardiac data

Segmentation of anatomical structures on 2D images of cardiac exams is necessary for performing 3D volumetric analysis, enabling the computation of parameters for diagnosing cardiovascular disease. In this work, we present robust algorithms to automatically segment cardiac imaging data and generate a volumetric anatomical reconstruction of a patient-specific heart model by propagating active contour output within a patient stack through a self-supervised learning model. Contour initializations are automatically generated, then output segmentations on sparse image slices are transferred and merged across a stack of images within the same heart data set during the segmentation process. We demonstrate whole-heart segmentation and compare the results with ground truth manual annotations. Additionally, we provide a framework to represent segmented heart data in the form of implicit surfaces, allowing interpolation operations to generate intermediary models of heart sections and volumes throughout the cardiac cycle and to estimate ejection fraction.

Real-time volume rendering with octree-based implicit surface representation

Recent breakthroughs in neural radiance fields have significantly advanced the field of novel view synthesis and 3D reconstruction from multi-view images. However, the prevalent neural volume rendering techniques often suffer from long rendering time and require extensive network training. To address these limitations, recent initiatives have explored explicit voxel representations of scenes to expedite training. Yet, they often fall short in delivering accurate geometric reconstructions due to a lack of effective 3D representation. In this paper, we propose an octree-based approach for the reconstruction of implicit surfaces from multi-view images. Leveraging an explicit, network-free data structure, our method substantially increases rendering speed, achieving real-time performance. Moreover, our reconstruction technique yields surfaces with quality comparable to state-of-the-art network-based learning methods. The source code and data can be downloaded from https://github.com/LaoChui999/Octree-VolSDF.

HISR: Hybrid Implicit Surface Representation for Photorealistic 3D Human Reconstruction

Neural reconstruction and rendering strategies have demonstrated state-of-the-art performances due, in part, to their ability to preserve high level shape details. Existing approaches, however, either represent objects as implicit surface functions or neural volumes and still struggle to recover shapes with heterogeneous materials, in particular human skin, hair or clothes. To this aim, we present a new hybrid implicit surface representation to model human shapes. This representation is composed of two surface layers that represent opaque and translucent regions on the clothed human body. We segment different regions automatically using visual cues and learn to reconstruct two signed distance functions (SDFs). We perform surface-based rendering on opaque regions (e.g., body, face, clothes) to preserve high-fidelity surface normals and volume rendering on translucent regions (e.g., hair). Experiments demonstrate that our approach obtains state-of-the-art results on 3D human reconstructions, and also shows competitive performances on other objects.

O^2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model

Occlusion is a common issue in 3D reconstruction from RGB-D videos, often blocking the complete reconstruction of objects and presenting an ongoing problem. In this paper, we propose a novel framework, empowered by a 2D diffusion-based in-painting model, to reconstruct complete surfaces for the hidden parts of objects. Specifically, we utilize a pre-trained diffusion model to fill in the hidden areas of 2D images. Then we use these in-painted images to optimize a neural implicit surface representation for each instance for 3D reconstruction. Since creating the in-painting masks needed for this process is tricky, we adopt a human-in-the-loop strategy that involves very little human engagement to generate high-quality masks. Moreover, some parts of objects can be totally hidden because the videos are usually shot from limited perspectives. To ensure recovering these invisible areas, we develop a cascaded network architecture for predicting signed distance field, making use of different frequency bands of positional encoding and maintaining overall smoothness. Besides the commonly used rendering loss, Eikonal loss, and silhouette loss, we adopt a CLIP-based semantic consistency loss to guide the surface from unseen camera angles. Experiments on ScanNet scenes show that our proposed framework achieves state-of-the-art accuracy and completeness in object-level reconstruction from scene-level RGB-D videos. Code: https://github.com/THU-LYJ-Lab/O2-Recon.

LISR: Learning Linear 3D Implicit Surface Representation Using Compactly Supported Radial Basis Functions

Implicit 3D surface reconstruction of an object from its partial and noisy 3D point cloud scan is the classical geometry processing and 3D computer vision problem. In the literature, various 3D shape representations have been developed, differing in memory efficiency and shape retrieval effectiveness, such as volumetric, parametric, and implicit surfaces. Radial basis functions provide memory-efficient parameterization of the implicit surface. However, we show that training a neural network using the mean squared error between the ground-truth implicit surface and the linear basis-based implicit surfaces does not converge to the global solution. In this work, we propose locally supported compact radial basis functions for a linear representation of the implicit surface. This representation enables us to generate 3D shapes with arbitrary topologies at any resolution due to their continuous nature. We then propose a neural network architecture for learning the linear implicit shape representation of the 3D surface of an object. We learn linear implicit shapes within a supervised learning framework using ground truth Signed-Distance Field (SDF) data for guidance. The classical strategies face difficulties in finding linear implicit shapes from a given 3D point cloud due to numerical issues (requires solving inverse of a large matrix) in basis and query point selection. The proposed approach achieves better Chamfer distance and comparable F-score than the state-of-the-art approach on the benchmark dataset. We also show the effectiveness of the proposed approach by using it for the 3D shape completion task.

Vox-Surf: Voxel-Based Implicit Surface Representation.

Virtual content creation and interaction play an important role in modern 3D applications. Recovering detailed 3D models from real scenes can significantly expand the scope of its applications and has been studied for decades in the computer vision and computer graphics community. In this work, we propose Vox-Surf, a voxel-based implicit surface representation. Our Vox-Surf divides the space into finite sparse voxels, where each voxel is a basic geometry unit that stores geometry and appearance information on its corner vertices. Due to the sparsity inherited from the voxel representation, Vox-Surf is suitable for almost any scene and can be easily trained end-to-end from multiple view images. We utilize a progressive training process to gradually cull out empty voxels and keep only valid voxels for further optimization, which greatly reduces the number of sample points and improves inference speed. Experiments show that our Vox-Surf representation can learn fine surface details and accurate colors with less memory and faster rendering than previous methods. The resulting fine voxels can also be considered as the bounding volumes for collision detection, which is useful in 3D interactions. We also show the potential application of Vox-Surf in scene editing and augmented reality. The source code is publicly available at https://github.com/zju3dv/Vox-Surf.

A GPU accelerated three-dimensional ghost cell method with an improved implicit surface representation for complex rigid or flexible boundary flows

This paper presents an efficient and reliable three-dimensional (3D) ghost cell method for complex rigid or flexible boundary flows. An improved implicit interface representation method is developed to treat the arbitrary complex interfaces including the thin or sharp boundaries. This method is robust and avoids the solving of a large dense matrix, and thus the computational efficiency is greatly enhanced. The ghost cell method with a unified eight-node interpolation scheme is adopted to enforce the no-slip boundary conditions on complex boundaries. To perform 3D high-resolution simulation on a desktop computer, a GPU parallel framework in CUDA environment is developed to significantly accelerate the computational efficiency of the present solver. The horizontal oscillations of a two-dimensional (2D) cylinder and a 3D sphere in the rest fluid are simulated to validate the accuracy and computational performance of the present model. Spatial accuracy of near second-order on the boundary treatment and good grid convergence are confirmed. Also, the present solver can achieve hundreds of acceleration ratios for 2D grids and thousands of acceleration ratios for 3D grids. Then, uniform flows around a 3D arbitrary rigid or flexible object such as a stationary cylindrical cylinder, a stationary square cylinder, an oscillating airfoil, and a swimming carangiform are simulated. The accuracy and capability of the present model for 3D complex geometries with large-amplitude movement or deformation is satisfactorily validated. Moreover, typical 3D wake structures are well captured.

Freetures: Localization in Signed Distance Function Maps

Localization of a robotic system within a previously mapped environment is important for reducing estimation drift and for reusing previously built maps. Existing techniques for geometry-based localization have focused on the description of local surface geometry, usually using pointclouds as the underlying representation. We propose a system for geometrybased localization that extracts features directly from an implicit surface representation: the Signed Distance Function (SDF). The SDF varies continuously through space, which allows the proposed system to extract and utilize features describing both surfaces and free-space. Through evaluations on public datasets, we demonstrate the flexibility of this approach, and show an increase in localization performance over state-of-the-art handcrafted surfaces-only descriptors. We achieve an average improvement of ~12% on an RGB-D dataset and ~18% on a LiDAR-based dataset. Finally, we demonstrate our system for localizing a LiDAR-equipped Micro Aerial Vehicle (MAV) within a previously built map of a search and rescue training ground. Our system, called freetures, is available as open source <formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex>$^{1}$</tex></formula> .

Neural Representation of Open Surfaces

AbstractNeural implicit surfaces have emerged as an effective, learnable representation for shapes of arbitrary topology. However, representing open surfaces remains a challenge. Different methods, such as unsigned distance fields (UDF), have been proposed to tackle this issue, but a general solution remains elusive. The generalized winding number (GWN), which is often used to distinguish interior points from exterior points of 3D shapes, is arguably the most promising approach. The GWN changes smoothly in regions where there is a hole in the surface, but it is discontinuous at points on the surface. Effectively, this means that it can be used in lieu of an implicit surface representation while providing information about holes, but, unfortunately, it does not provide information about the distance to the surface necessary for e.g. ray tracing, and special care must be taken when implementing surface reconstruction. Therefore, we introduce the semi‐signed distance field (SSDF) representation which comprises both the GWN and the surface distance. We compare the GWN and SSDF representations for the applications of surface reconstruction, interpolation, reconstruction from partial data, and latent vector analysis using two very different data sets. We find that both the GWN and SSDF are well suited for neural representation of open surfaces.

Three-dimensional shape generation via variational autoencoder generative adversarial network with signed distance function

&lt;span lang="EN-US"&gt;Mesh-based 3-dimensional (3D) shape generation from a 2-dimensional (2D) image using a convolution neural network (CNN) framework is an open problem in the computer graphics and vision domains. Most existing CNN-based frameworks lack robust algorithms that can scale well without combining different shape parts. Also, most CNN-based algorithms lack suitable 3D data representations that can fit into CNN without modification(s) to produce high-quality 3D shapes. This paper presents an approach that integrates a variational autoencoder (VAE) and a generative adversarial network (GAN) called 3 dimensional variational autoencoder signed distance function generative adversarial network (3D-VAE-SDFGAN) to create a 3D shape from a 2D image that considerably improves scalability and visual quality. The proposed method only feeds a single 2D image into the network to produce a mesh-based 3D shape. The network encodes a 2D image of the 3D object into the latent representations, and implicit surface representations of 3D objects corresponding to those 2D images are subsequently generated. Hence, a signed distance function (SDF) is proposed to maintain object inside-outside information in the implicit surface representation. Polygon mesh surfaces are then produced using the marching cubes algorithm. The ShapeNet dataset was used in the experiments to evaluate the proposed 3D-VAE-SDFGAN. The experimental results show that 3D-VAE-SDFGAN outperforms other state-of-the-art models.&lt;/span&gt;

Learning neural implicit representations with surface signal parameterizations

Neural implicit surface representations have recently emerged as popular alternative to explicit 3D object encodings, such as polygonal meshes, tabulated points, or voxels. While significant work has improved the geometric fidelity of these representations, much less attention is given to their final appearance. Traditional explicit object representations commonly couple the 3D shape data with auxiliary surface-mapped image data, such as diffuse color textures and fine-scale geometric details in normal maps that typically require a mapping of the 3D surface onto a plane, i.e., a surface parameterization; implicit representations, on the other hand, cannot be easily textured due to lack of configurable surface parameterization. Inspired by this digital content authoring methodology, we design a neural network architecture that implicitly encodes the underlying surface parameterization suitable for appearance data. As such, our model remains compatible with existing mesh-based digital content with appearance data. Motivated by recent work that overfits compact networks to individual 3D objects, we present a new weight-encoded neural implicit representation that extends the capability of neural implicit surfaces to enable various common and important applications of texture mapping. Our method outperforms reasonable baselines and state-of-the-art alternatives.

3D Shape Generation via Variational Autoencoder with Signed Distance Function Relativistic Average Generative Adversarial Network

3D shape generation is widely applied in various industries to create, visualize, and analyse complex data, designs, and simulations. Typically, 3D shape generation uses a large dataset of 3D shapes as the input. This paper proposes a variational autoencoder with a signed distance function relativistic average generative adversarial network, referred to as 3D-VAE-SDFRaGAN, for 3D shape generation from 2D input images. Both the generative adversarial network (GAN) and variational autoencoder (VAE) algorithms are typical algorithms used to generate realistic 3D shapes. However, it is very challenging to train a stable 3D shape generation model using VAE-GAN. This paper proposes an efficient approach to stabilize the training process of VAE-GAN to generate high-quality 3D shapes. A 3D mesh-based shape is first generated using a 3D signed distance function representation by feeding a single 2D image into a 3D-VAE-SDFRaGAN network. The signed distance function is used to maintain inside–outside information in the implicit surface representation. In addition, a relativistic average discriminator loss function is employed as the training loss function. The polygon mesh surfaces are then produced via the marching cubes algorithm. The proposed 3D-VAE-SDFRaGAN is evaluated with the ShapeNet dataset. The experimental results indicate a notable enhancement in the qualitative performance, as evidenced by the visual comparison of the generated samples, as well as the quantitative performance evaluation using the chamfer distance metric. The proposed approach achieves an average chamfer distance score of 0.578, demonstrating superior performance compared to existing state-of-the-art models.

Multiview Textured Mesh Recovery by Differentiable Rendering

Although having achieved the promising results on shape and color recovery through self-supervision, the multi-layer perceptrons-based methods usually suffer from heavy computational cost on learning the deep implicit surface representation. Since rendering each pixel requires a forward network inference, it is very computational intensive to synthesize a whole image. To tackle these challenges, we propose an effective coarse-to-fine approach to recover the textured mesh from multi-views in this paper. Specifically, a differentiable Poisson Solver is employed to represent the object's shape, which is able to produce topology-agnostic and watertight surfaces. To account for depth information, we optimize the shape geometry by minimizing the differences between the rendered mesh and the predicted depth from multi-view stereo. In contrast to the implicit neural representation on shape and color, we introduce a physically based inverse rendering scheme to jointly estimate the environment lighting and object's reflectance, which is able to render the high resolution image at real-time. The texture of the reconstructed mesh is interpolated from a learnable dense texture grid. We have conducted the extensive experiments on several multi-view stereo datasets, whose promising results demonstrate the efficacy of our proposed approach. The code is available at https://github.com/l1346792580123/diff.

Geometric advection and its application in the emulation of high aspect ratio structures

The ability to simulate the processes required to fabricate advanced microelectronic structures, commonly referred to as process technology computer aided design (TCAD), is essential for the semiconductor industry. It aids in the design and development of modern integrated circuits at low cost. Ongoing demands of high efficiency and simplicity lead to the development of process emulation techniques, which describe the effect of a fabrication step by using only the geometric properties gathered from experimental data or extracted from physical simulations. However, the accuracy of current emulation models is limited by their strong reliance on explicit surface representations, while traditional algorithms employing implicit surfaces, such as the level set, can frequently suffer from numerical limitations. A novel geometric advection algorithm for microelectronic process emulation is presented, which advances a level set surface over large distances in a single simulation step. It combines the advantages of explicit surface representations, such as fast access to surface data and efficient sorting, and the robustness of implicit surface representations. Furthermore, calculations are performed directly in the level set avoiding costly conversions to explicit surfaces. The performance and applicability of the algorithm to modern processes is demonstrated by emulating the fabrication of several high aspect ratio structures. Due to the almost universal applicability and computational efficiency of the presented geometric advection algorithm, it provides an entirely new way of emulating microelectronic fabrication processes and presents a promising alternative to long-standing and commonly used techniques.

Frictional Contact on Smooth Elastic Solids

Frictional contact between deformable elastic objects remains a difficult simulation problem in computer graphics. Traditionally, contact has been resolved using sophisticated collision detection schemes and methods that build on the assumption that contact happens between polygons. While polygonal surfaces are an efficient representation for solids, they lack some intrinsic properties that are important for contact resolution. Generally, polygonal surfaces are not equipped with an intrinsic inside and outside partitioning or a smooth distance field close to the surface. Here we propose a new method for resolving frictional contacts against deforming implicit surface representations that addresses these problems. We augment a moving least squares (MLS) implicit surface formulation with a local kernel for resolving contacts, and develop a simple parallel transport approximation to enable transfer of frictional impulses. Our variational formulation of dynamics and elasticity enables us to naturally include contact constraints, which are resolved as one Newton-Raphson solve with linear inequality constraints. We extend this formulation by forwarding friction impulses from one time step to the next, used as external forces in the elasticity solve. This maintains the decoupling of friction from elasticity thus allowing for different solvers to be used in each step. In addition, we develop a variation of staggered projections, that relies solely on a non-linear optimization without constraints and does not require a discretization of the friction cone. Our results compare favorably to a popular industrial elasticity solver (used for visual effects), as well as recent academic work in frictional contact, both of which rely on polygons for contact resolution. We present examples of coupling between rigid bodies, cloth and elastic solids.

Fast Ray Tracing of Scale‐Invariant Integral Surfaces

AbstractScale‐invariant integral surfaces, which are implicit representations of surfaces, provide a way to define smooth surfaces from skeletons with prescribed radii defined at their vertices. We introduce a new rendering pipeline allowing to visualize such surfaces in real‐time. We rely on the distance to skeleton to define a sampling strategy along the camera rays, dividing each ray into sub‐intervals. The proposed strategy is chosen to capture main field variations. Resulting intervals are processed iteratively, relying on two main ingredients; quadratic interpolation and field mapping, to an approximate squared homothetic distance. The first provides efficient root finding while the second increases the precision of the interpolation, and the combination of both results in an efficient processing routine. Finally, we present a GPU implementation that relies on a dynamic data‐structure in order to efficiently generate the intervals along the ray. This data‐structure also serves as an acceleration structure that allows constant time access to the primitives of interest during the processing of a given ray.

Point Cloud Registration Using Virtual Interest Points from Macaulay’s Resultant of Quadric Surfaces

A novel formulation called Virtual Interest Point is presented and used to register point clouds. An implicit quadric surface representation is first used to model the point cloud segments. Macaulay’s resultant then provides the intersection of three such quadrics, which forms a virtual interest point (VIP). A unique feature descriptor for each VIP is computed, and correspondences in descriptor space are established to compute the rigid transformation to register two point clouds. Each step in the process is designed to consider robustness to noise and data density variations, as well as computational efficiency. Experiments were performed on 12 data sets, collected with a variety of range sensors, to characterize robustness to noise, data density variation, and computational efficiency. The data sets were extracted from both natural scenes, including plants and rocks, and indoor architectural scenes, such as cluttered offices and laboratories. Similarly, several 3D models were tested for registration to demonstrate the generality of the technique. The proposed method significantly outperformed a variety of alternative state-of-the-art approaches, such as 2.5D SIFT-based RANSAC method, Super 4-Point Congruent Sets and Super Generalized 4PCS, and the Go-ICP method in registering overlapping point clouds with both a higher success rate and reduced computational cost.

Vox2Surf: Implicit Surface Reconstruction from Volumetric Data.

Surface reconstruction from volumetric T1-weighted and T2-weighted images is a time-consuming multi-step process that often involves careful parameter fine-tuning, hindering a more wide-spread utilization of surface-based analysis particularly in large-scale studies. In this work, we propose a fast surface reconstruction method that is based on directly learning a continuous-valued signed distance function (SDF) as implicit surface representation. This continuous representation implicitly encodes the boundary of the surface as the zero isosurface. Given the predicted SDF, the target 3D surface is reconstructed by applying the marching cubes algorithm. Our implicit reconstruction method concurrently predicts the surfaces of the brain parenchyma, the white matter and pial surfaces, the subcortical structures, and the ventricles. Evaluation based on data from the Human Connectome Project indicates that surface reconstruction of a total of 22 cortical and subcortical structures can be completed in less than 20 min.

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

We present a learning-based approach to reconstructing high-resolution three-dimensional (3D) shapes with detailed geometry and high-fidelity textures. Albeit extensively studied, algorithms for 3D reconstruction from multi-view depth-and-color (RGB-D) scans are still prone to measurement noise and occlusions; limited scanning or capturing angles also often lead to incomplete reconstructions. Propelled by recent advances in 3D deep learning techniques, in this paper, we introduce a novel computation- and memory-efficient cascaded 3D convolutional network architecture, which learns to reconstruct implicit surface representations as well as the corresponding color information from noisy and imperfect RGB-D maps. The proposed 3D neural network performs reconstruction in a progressive and coarse-to-fine manner, achieving unprecedented output resolution and fidelity. Meanwhile, an algorithm for end-to-end training of the proposed cascaded structure is developed. We further introduce Human10, a newly created dataset containing both detailed and textured full-body reconstructions as well as corresponding raw RGB-D scans of 10 subjects. Qualitative and quantitative experimental results on both synthetic and real-world datasets demonstrate that the presented approach outperforms existing state-of-the-art work regarding visual quality and accuracy of reconstructed models.

Simulation of finishing operations based on an implicit representation of surfaces

The manufactured surface textures after machining operations may require the use of an abrasion operation in order to meet expected requirements. In this context, the simulation of the entire manufacturing process from machining to abrasion finishing could significantly improve the parameters involved in manufacturing. The simulation of the abrasion requires to handle a high number of interactions between the workpiece and abrasive grains as well as complex mechanical behavior during material removal. Considering numerical and physical constraints, a novel model for abrasion simulations based on an implicit surface representation has been developed which allows macroscopic simulations of surface topography resulting from abrasion. The surface is embedded in a scalar field, so that the removal action occurs throughout the modification of the volumetric data rather than an explicit interaction with the surface. In this article, the implicit model is enriched in order to be able to integrate the surface topography of the previous machining operation. Thus, the simulation of the resulting surface topography for an entire manufacturing process has been performed successfully. Several experiments have been carried out and analyzed in order to evaluate the proposed model. The comparison between simulations and experimental results validates the consistency of the model regarding real abrasion finishing processes.