Radiance Field Research Articles

Spiking-NeRF: Spiking Neural Network for Energy-Efficient Neural Rendering

Artificial Neural Networks (ANNs) have achieved remarkable performance in many artificial intelligence tasks. As the application scenarios become more sophisticated, the computation and energy consumption of ANNs are also constantly increasing, which poses a challenge for deploying ANNs on energy-constrained devices. Spiking Neural Networks (SNNs) provide a promising solution to build energy-efficiency neural networks. However, the current training methods of SNNs cannot output values as precise as ANNs. This limits the applications of SNNs to relatively simple image classification tasks. In this article, we extend the application of SNNs to neural rendering tasks and propose an energy-efficient spiking neural rendering model, called Spiking-NeRF (Spiking Neural Radiance Fields). We first analyze the ANN-to-SNN conversion theory and propose an output scheme for SNNs to obtain the precise scene property values. Then we customize the parameter normalization method for the special network architecture of neural rendering. Furthermore, we present an early termination strategy (ETS) based on the discrete nature of spikes to reduce energy consumption. We evaluate the performance of Spiking-NeRF on both realistic and synthetic scenes. Experimental results show that Spiking-NeRF can achieve comparable rendering performance to ANN-based NeRF with up to \(2.27\times\) energy reduction.

TVNeRF: Improving few-view neural volume rendering with total variation maximization

Neural Radiance Fields (NeRF) has profoundly impacted few-shot novel view synthesis and 3D reconstruction with its innovative rendering techniques and high-quality output. However, synthesizing novel views from sparse inputs remains challenging for NeRF. With limited view inputs, NeRF tends to overfit the available views, leading to reconstruction discrepancies and artifacts. To address this issue, we propose TVNeRF, which introduces two complementary regularization methods. By maximizing the total variation of unseen rays, we control the density distribution of ray sampled points to prevent excessive sampling in non-critical areas, thereby reducing artifacts and floaters. Concurrently, minimizing the accumulated opacity of unseen rays allows for the effective selection of important rays and sampled points without additional computational overhead, while also mitigating potential over-regularization effects. These combined strategies not only prevent overfitting but also achieve effects similar to the Least Absolute Shrinkage and Selection Operator (LASSO) regularization. Moreover, TVNeRF achieves these improvements without the need for pre-trained models or prior knowledge. By utilizing these straightforward yet effective regularization techniques, TVNeRF can deliver results comparable to state-of-the-art models while maintaining computational efficiency. This approach highlights that substantial improvements in NeRF performance can be realized through targeted regularization methods, effectively addressing the challenges posed by sparse view synthesis.

Point'n Move: Interactive scene object manipulation on Gaussian splatting radiance fields

AbstractThe authors propose Point'n Move, a method that achieves interactive scene object manipulation with exposed region inpainting. Interactivity here further comes from intuitive object selection and real‐time editing. To achieve this, Gaussian Splatting Radiance Field is adopted as the scene representation and its explicit nature and speed advantage are fully leveraged. Its explicit representation formulation allows to devise a 2D prompt points to 3D masks dual‐stage self‐prompting segmentation algorithm, perform mask refinement and merging, minimize changes, and provide good initialization for scene inpainting and perform editing in real‐time without per‐editing training; all lead to superior quality and performance. The method was tested by editing both forward‐facing and 360 scenes. The method is also compared against existing methods, showing superior quality despite being more capable and having a speed advantage.

A Diffusion Approach to Radiance Field Relighting using Multi‐Illumination Synthesis

AbstractRelighting radiance fields is severely underconstrained for multi‐view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single‐illumination data by exploiting priors extracted from 2D image diffusion models. We first fine‐tune a 2D diffusion model on a multi‐illumination dataset conditioned by light direction, allowing us to augment a single‐illumination capture into a realistic – but possibly inconsistent – multi‐illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low‐frequency lighting, we represent appearance with a multi‐layer perceptron parameterized on light direction. To enforce multi‐view consistency and overcome inaccuracies we optimize a per‐image auxiliary feature vector. We show results on synthetic and real multi‐view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes.

Improving Neural Radiance Fields Using Near-Surface Sampling with Point Cloud Generation

Neural radiance field (NeRF) is an emerging view synthesis method that samples points in a three-dimensional (3D) space and estimates their existence and color probabilities. The disadvantage of NeRF is that it requires a long training time since it samples many 3D points. In addition, if one samples points from occluded regions or in the space where an object is unlikely to exist, the rendering quality of NeRF can be degraded. These issues can be solved by estimating the geometry of 3D scene. This paper proposes a near-surface sampling framework to improve the rendering quality of NeRF. To this end, the proposed method estimates the surface of a 3D object using depth images of the training set and performs sampling only near the estimated surface. To obtain depth information on a novel view, the paper proposes a 3D point cloud generation method and a simple refining method for projected depth from a point cloud. Experimental results show that the proposed near-surface sampling NeRF framework can significantly improve the rendering quality, compared to the original NeRF and three different state-of-the-art NeRF methods. In addition, one can significantly accelerate the training time of a NeRF model with the proposed near-surface sampling framework.

Simplicits: Mesh-Free, Geometry-Agnostic Elastic Simulation

The proliferation of 3D representations, from explicit meshes to implicit neural fields and more, motivates the need for simulators agnostic to representation. We present a data-, mesh-, and grid-free solution for elastic simulation for any object in any geometric representation undergoing large, nonlinear deformations. We note that every standard geometric representation can be reduced to an occupancy function queried at any point in space, and we define a simulator atop this common interface. For each object, we fit a small implicit neural network encoding spatially varying weights that act as a reduced deformation basis. These weights are trained to learn physically significant motions in the object via random perturbations. Our loss ensures we find a weight-space basis that best minimizes deformation energy by stochastically evaluating elastic energies through Monte Carlo sampling of the deformation volume. At runtime, we simulate in the reduced basis and sample the deformations back to the original domain. Our experiments demonstrate the versatility, accuracy, and speed of this approach on data including signed distance functions, point clouds, neural primitives, tomography scans, radiance fields, Gaussian splats, surface meshes, and volume meshes, as well as showing a variety of material energies, contact models, and time integration schemes.

NeuralTO: Neural Reconstruction and View Synthesis of Translucent Objects

Learning from multi-view images using neural implicit signed distance functions shows impressive performance on 3D Reconstruction of opaque objects. However, existing methods struggle to reconstruct accurate geometry when applied to translucent objects due to the non-negligible bias in their rendering function. To address the inaccuracies in the existing model, we have reparameterized the density function of the neural radiance field by incorporating an estimated constant extinction coefficient. This modification forms the basis of our innovative framework, which is geared towards highfidelity surface reconstruction and the novel-view synthesis of translucent objects. Our framework contains two stages. In the reconstruction stage, we introduce a novel weight function to achieve accurate surface geometry reconstruction. Following the recovery of geometry, the second phase involves learning the distinct scattering properties of the participating media to enhance rendering. A comprehensive dataset, comprising both synthetic and real translucent objects, has been built for conducting extensive experiments. Experiments reveal that our method outperforms existing approaches in terms of reconstruction and novel-view synthesis.

Bilateral Guided Radiance Field Processing

Neural Radiance Fields (NeRF) achieves unprecedented performance in synthesizing novel view synthesis, utilizing multi-view consistency. When capturing multiple inputs, image signal processing (ISP) in modern cameras will independently enhance them, including exposure adjustment, color correction, local tone mapping, etc. While these processings greatly improve image quality, they often break the multi-view consistency assumption, leading to "floaters" in the reconstructed radiance fields. To address this concern without compromising visual aesthetics, we aim to first disentangle the enhancement by ISP at the NeRF training stage and re-apply user-desired enhancements to the reconstructed radiance fields at the finishing stage. Furthermore, to make the re-applied enhancements consistent between novel views, we need to perform imaging signal processing in 3D space (i.e. "3D ISP"). For this goal, we adopt the bilateral grid, a locally-affine model, as a generalized representation of ISP processing. Specifically, we optimize per-view 3D bilateral grids with radiance fields to approximate the effects of camera pipelines for each input view. To achieve user-adjustable 3D finishing, we propose to learn a low-rank 4D bilateral grid from a given single view edit, lifting photo enhancements to the whole 3D scene. We demonstrate our approach can boost the visual quality of novel view synthesis by effectively removing floaters and performing enhancements from user retouching. The source code and our data are available at: https://bilarfpro.github.io.

SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration

Recent techniques for real-time view synthesis have rapidly advanced in fidelity and speed, and modern methods are capable of rendering near-photorealistic scenes at interactive frame rates. At the same time, a tension has arisen between explicit scene representations amenable to rasterization and neural fields built on ray marching, with state-of-the-art instances of the latter surpassing the former in quality while being prohibitively expensive for real-time applications. We introduce SMERF, a view synthesis approach that achieves state-of-the-art accuracy among real-time methods on large scenes with footprints up to 300 m 2 at a volumetric resolution of 3.5 mm 3 . Our method is built upon two primary contributions: a hierarchical model partitioning scheme, which increases model capacity while constraining compute and memory consumption, and a distillation training strategy that simultaneously yields high fidelity and internal consistency. Our method enables full six degrees of freedom navigation in a web browser and renders in real-time on commodity smartphones and laptops. Extensive experiments show that our method exceeds the state-of-the-art in real-time novel view synthesis by 0.78 dB on standard benchmarks and 1.78 dB on large scenes, renders frames three orders of magnitude faster than state-of-the-art radiance field models, and achieves real-time performance across a wide variety of commodity devices, including smartphones. We encourage readers to explore these models interactively at our project website: https://smerf-3d.github.io.

FVDB : A Deep-Learning Framework for Sparse, Large Scale, and High Performance Spatial Intelligence

We present f VDB, a novel GPU-optimized framework for deep learning on large-scale 3D data. f VDB provides a complete set of differentiable primitives to build deep learning architectures for common tasks in 3D learning such as convolution, pooling, attention, ray-tracing, meshing, etc. f VDB simultaneously provides a much larger feature set (primitives and operators) than established frameworks with no loss in efficiency: our operators match or exceed the performance of other frameworks with narrower scope. Furthermore, f VDB can process datasets with much larger footprint and spatial resolution than prior works, while providing a competitive memory footprint on small inputs. To achieve this combination of versatility and performance, f VDB relies on a single novel VDB index grid acceleration structure paired with several key innovations including GPU accelerated sparse grid construction, convolution using tensorcores, fast ray tracing kernels using a Hierarchical Digital Differential Analyzer algorithm (HDDA), and jagged tensors. Our framework is fully integrated with PyTorch enabling interoperability with existing pipelines, and we demonstrate its effectiveness on a number of representative tasks such as large-scale point-cloud segmentation, high resolution 3D generative modeling, unbounded scale Neural Radiance Fields, and large-scale point cloud reconstruction.

Temporal residual neural radiance fields for monocular video dynamic human body reconstruction

Temporal residual neural radiance fields for monocular video dynamic human body reconstruction

IW-NeRF: Using Implicit Watermarks to Protect the Copyright of Neural Radiation Fields

The neural radiance field (NeRF) has demonstrated significant advancements in computer vision. However, the training process for NeRF models necessitates extensive computational resources and ample training data. In the event of unauthorized usage or theft of the model, substantial losses can be incurred by the copyright holder. To address this concern, we present a novel algorithm that leverages the implicit neural representation (INR) watermarking technique to safeguard NeRF model copyrights. By encoding the watermark information implicitly, we integrate its parameters into the NeRF model’s network using a unique key. Through this key, the copyright owner can extract the embedded watermarks from the NeRF model for ownership verification. To the best of our knowledge, this is the pioneering implementation of INR watermarking for the protection of NeRF model copyrights. Our experimental results substantiate that our approach not only offers robustness and preserves high-quality 3D reconstructions but also ensures the flawless (100%) extraction of watermark content, thereby effectively securing the copyright of the NeRF model.

Constraining the Geometry of NeRFs for Accurate DSM Generation from Multi-View Satellite Images

Neural Radiance Fields (NeRFs) are an emerging approach to 3D reconstruction that use neural networks to reconstruct scenes. However, its applications for multi-view satellite photogrammetry, which aim to reconstruct the Earth’s surface, struggle to acquire accurate digital surface models (DSMs). To address this issue, a novel framework, Geometric Constrained Neural Radiance Field (GC-NeRF) tailored for multi-view satellite photogrammetry, is proposed. GC-NeRF achieves higher DSM accuracy from multi-view satellite images. The key point of this approach is a geometric loss term, which constrains the scene geometry by making the scene surface thinner. The geometric loss term alongside z-axis scene stretching and multi-view DSM fusion strategies greatly improve the accuracy of generated DSMs. During training, bundle-adjustment-refined satellite camera models are used to cast rays through the scene. To avoid the additional input of altitude bounds described in previous works, the sparse point cloud resulting from the bundle adjustment is converted to an occupancy grid to guide the ray sampling. Experiments on WorldView-3 images indicate GC-NeRF’s superiority in accurate DSM generation from multi-view satellite images.

Efficient neural implicit representation for 3D human reconstruction

High-fidelity digital human representations are increasingly in demand in the digital world, particularly for interactive telepresence, AR/VR, 3D graphics, and the rapidly evolving metaverse. Even though they work well in small spaces, conventional methods for reconstructing 3D human motion frequently require the use of expensive hardware and have high processing costs. This study presents HumanAvatar, an innovative approach that efficiently reconstructs precise human avatars from monocular video sources. At the core of our methodology, we integrate the pre-trained HuMoR, a model celebrated for its proficiency in human motion estimation. This is adeptly fused with the cutting-edge neural radiance field technology, Instant-NGP, and the state-of-the-art articulated model, Fast-SNARF, to enhance the reconstruction fidelity and speed. By combining these two technologies, a system is created that can render quickly and effectively while also providing estimation of human pose parameters that are unmatched in accuracy. We have enhanced our system with an advanced posture-sensitive space reduction technique, which optimally balances rendering quality with computational efficiency. In our detailed experimental analysis using both artificial and real-world monocular videos, we establish the advanced performance of our approach. HumanAvatar consistently equals or surpasses contemporary leading-edge reconstruction techniques in quality. Furthermore, it achieves these complex reconstructions in minutes, a fraction of the time typically required by existing methods. Our models achieve a training speed that is 110× faster than that of State-of-The-Art (SoTA) NeRF-based models. Our technique performs noticeably better than SoTA dynamic human NeRF methods if given an identical runtime limit. HumanAvatar can provide effective visuals after only 30 s of training. Please visit https://github.com/HZXu-526/Human-Avatar for further demo results and code.

TraM‐NeRF: Tracing Mirror and Near‐Perfect Specular Reflections Through Neural Radiance Fields

AbstractImplicit representations like neural radiance fields (NeRF) showed impressive results for photorealistic rendering of complex scenes with fine details. However, ideal or near‐perfectly specular reflecting objects such as mirrors, which are often encountered in various indoor scenes, impose ambiguities and inconsistencies in the representation of the re‐constructed scene leading to severe artifacts in the synthesized renderings. In this paper, we present a novel reflection tracing method tailored for the involved volume rendering within NeRF that takes these mirror‐like objects into account while avoiding the cost of straightforward but expensive extensions through standard path tracing. By explicitly modelling the reflection behaviour using physically plausible materials and estimating the reflected radiance with Monte‐Carlo methods within the volume rendering formulation, we derive efficient strategies for importance sampling and the transmittance computation along rays from only few samples. We show that our novel method enables the training of consistent representations of such challenging scenes and achieves superior results in comparison to previous state‐of‐the‐art approaches.

Instruct Pix-to-3D: Instructional 3D object generation from a single image

We study a challenging task, creating the instructional 3D object based on a single image and human instruction. Although existing methods utilize pre-trained text-to-image diffusion models to optimize Neural Radiance Fields (NeRF) and achieve remarkable results, they are inherently challenged in addressing this task due to two inherent obstacles. First, the written instruction is mismatched or uncorrelated with target images in the distribution of pre-trained text-to-image diffusion models, which leads to insufficient supervision. Another is editing 3D content starting from a single image, which is an ill-posed problem as the image only provides partial information. To tackle the difficulties, we propose an Instruct Pixel-to-3D framework, which exploits the supervisory signals from the given instructions and integrates the domain-specific knowledge drawn from 2D and 3D diffusion models. Specifically, to address the first challenge, we propose instruction guidance to facilitate meticulous image editing using fine-grained instructions and regulate the novel views of 3D objects using positional prompts. To tackle the second difficulty, we propose diffusion guidance for 3D generation, which optimizes the NeRF employing dual guidances: a 2D diffusion guidance derived from the score distillation sampling (SDS) loss and a 3D diffusion guidance embodied by the Zero-1-to-3 loss. Moreover, Instruct Pix-to-3D adopts a coarse-to-fine training strategy aimed at mitigating the complexities inherent in reconstructing scenes from sparse viewpoints. Qualitative and quantitative experiments on the DTU dataset and in-the-wild images demonstrate that our approach could synthesize high-quality and high-fidelity 3D content based on a single image and human instruction.

Points2NeRF: Generating Neural Radiance Fields from 3D point cloud

Neural Radiance Fields (NeRFs) offers a state-of-the-art quality in synthesizing novel views of complex 3D scenes from a small subset of base images. For NeRFs to perform optimally, the registration of base images has to follow certain assumptions, including maintaining a constant distance between the camera and the object. We can address this limitation by training NeRFs with 3D point clouds instead of images, yet a straightforward substitution is impossible due to the sparsity of 3D clouds in the under-sampled regions, which leads to incomplete reconstruction output by NeRFs. To solve this problem, here we propose an auto-encoder-based architecture that leverages a hypernetwork paradigm to transfer 3D points with the associated color values through a lower-dimensional latent space and generate weights of NeRF model. This way, we can accommodate the sparsity of 3D point clouds and fully exploit the potential of point cloud data. As a side benefit, our method offers an implicit way of representing 3D scenes and objects that can be employed to condition NeRFs and hence generalize the models beyond objects seen during training. The empirical evaluation confirms the advantages of our method over conventional NeRFs and proves its superiority in practical applications.

Reflective signatures of unresolved objects.

We present a method for identifying, classifying, and distinguishing unresolved reflective objects. A forward model is developed to predict how the radiance field from a specular surface will be angularly distributed and how samples detected from that field can be used to infer surface profile characteristics. We present lab studies to validate the forward model and demonstrate unresolved object identification and classification. We demonstrate unresolved specular object identification for a 35 mm target at a 4.6 km and 50 mm targets at a 27.6 km range with a 28 cm aperture telescope. Preliminary observations for specular signatures of drones and helicopters are also presented.

Integration of 3D Gaussian Splatting and Neural Radiance Fields in Virtual Reality Fire Fighting

Neural radiance fields (NeRFs) and 3D Gaussian splatting have emerged as promising 3D reconstruction techniques recently. However, their application in virtual reality (VR), particularly in firefighting training, remains underexplored. We present an innovative VR firefighting simulation system based on 3D Gaussian Splatting technology. Leveraging these techniques, we successfully reconstruct realistic physical environments. By integrating the Unity3D game engine with head-mounted displays (HMDs), we created and presented immersive virtual fire scenes. Our system incorporates NeRF technology to generate highly realistic models of firefighting equipment. Users can freely navigate and interact with fire within the virtual fire scenarios, enhancing immersion and engagement. Moreover, by utilizing the Photon PUN2 networking framework, our system enables multi-user collaboration on firefighting tasks, improving training effectiveness and fostering teamwork and communication skills. Through experiments and surveys, it is demonstrated that the proposed VR framework enhances user experience and holds promises for improving the effectiveness of firefighting training.

Portrait relighting for 3D light-field display based on radiance fields

Relighting a portrait scene from a photographs or video requires accounting for high-order light transport effects and accurate estimation of portrait scene geometry, specular reflections, shadows, and subsurface scattering. A delicate manipulation of the lighting can be performed while keeping the scene albedo and geometry unaltered. A radiance field-based approach to light-field rendering of portrait scenes is proposed in this work. The method requires only a mobile phone with a flash and an inexpensive polarization foil for data acquisition. Two groups of portrait images are captured using cross-polarized light and parallel polarized light. Subsequently, two multi-layer perceptrons (MLPs) are employed to extract the material information (Albedo, Normals, Specular) of the portrait individually. The extracted material information, along with an ambient light map, is utilized in physically-based rendering to generate portrait images. The final result is optimized using a coarse-to-fine optimization scheme. Finally, a light-field image is generated through light-field encoding. Our approach achieves higher quality light-field portrait scenes with controllable illumination, realistic shadows, and subsurface scattering (as shown in Fig. 1). The method we propose can be applied to small-sized light-field displays to enhance the lighting quality of portrait reconstructions, and lay the foundation for future applications in light-field communication and related fields.