Radiance Field Research Articles

NeRFtrinsic Four: An end-to-end trainable NeRF jointly optimizing diverse intrinsic and extrinsic camera parameters

Novel view synthesis using neural radiance fields (NeRF) is the state-of-the-art technique for generating high-quality images from novel viewpoints. Existing methods require a priori knowledge about extrinsic and intrinsic camera parameters. This limits their applicability to synthetic scenes, or real-world scenarios with the necessity of a preprocessing step. Current research on the joint optimization of camera parameters and NeRF focuses on refining noisy extrinsic camera parameters and often relies on the preprocessing of intrinsic camera parameters. Further approaches are limited to cover only one single camera intrinsic. To address these limitations, we propose a novel end-to-end trainable approach called NeRFtrinsic Four. We utilize Gaussian Fourier features to estimate extrinsic camera parameters and dynamically predict varying intrinsic camera parameters through the supervision of the projection error. Our approach outperforms existing joint optimization methods on LLFF and BLEFF. In addition to these existing datasets, we introduce a new dataset called iFF with varying intrinsic camera parameters. NeRFtrinsic Four is a step forward in joint optimization NeRF-based view synthesis and enables more realistic and flexible rendering in real-world scenarios with varying camera parameters.

LTM-NeRF: Embedding 3D Local Tone Mapping in HDR Neural Radiance Field.

Recent advances in Neural Radiance Fields (NeRF) have provided a new geometric primitive for novel view synthesis. High Dynamic Range NeRF (HDR NeRF) can render novel views with a higher dynamic range. However, effectively displaying the scene contents of HDR NeRF on diverse devices with limited dynamic range poses a significant challenge. To address this, we present LTM-NeRF, a method designed to recover HDR NeRF and support 3D local tone mapping. LTM-NeRF allows for the synthesis of HDR views, tone-mapped views, and LDR views under different exposure settings, using only the multi-view multi-exposure LDR inputs for supervision. Specifically, we propose a differentiable Camera Response Function (CRF) module for HDR NeRF reconstruction, globally mapping the scene's HDR radiance to LDR pixels. Moreover, we introduce a Neural Exposure Field (NeEF) to represent the spatially varying exposure time of an HDR NeRF to achieve 3D local tone mapping, for compatibility with various displays. Comprehensive experiments demonstrate that our method can not only synthesize HDR views and exposure-varying LDR views accurately but also render locally tone-mapped views naturally.

Two-dimensional P1 approximation (P1-2D) for the Description of the Radiant Field on Cylindrical Solar Photocatalytic Reactors

The local volumetric rate of photon absorption (LVRPA) was formulated by solving the radiative transfer equation (RTE) in polar coordinates with the P1 approximation approach (P1-2D) for the description of the radiant field in cylindrical solar photocatalytic reactors. A general expression of the LVRPA was formulated that can be employed on cylindrical photocatalytic reactors with an incident radiation constant along the reactor length. CPC and tubular photocatalytic reactors were used as reactor models and Lambert's cosine law (irradiance) was considered when using the boundary conditions. Simulations were carried out using the commercial TiO2-P25, its optical properties taken from the literature. The LVRPA was found to decrease exponentially from the reactor wall to its center. literature rate of photon absorption per unit of reactor length (VRPA/H) increased exponentially with the catalyst loading until a value where no significant increase was observed and was found to increase with reactor radius, information that agrees with the literature. The optimum catalyst loading with the CPC reactor was about 0.364 g/L with a reactor radius equal to 1.65 cm similar to that found in the literature when using the six-flux model in two dimensions (SFM-2D). The apparent optical thickness τ_App1 newly formulated with the P1 approximation was introduced for optimization purposes and was found more reliable than the optical thickness τ. This parameter not only removes the dependence of the optimum catalyst loading on the reactor's radius but also its dependence on catalyst albedo. τ_App1 was found about 9.73 and 14.6 for CPC and tubular reactors respectively and provides the optimum catalyst loading and the reactor radius that optimize the radiation absorption inside both reactors.

Unmanned Aerial Vehicle-Neural Radiance Field (UAV-NeRF): Learning Multiview Drone Three-Dimensional Reconstruction with Neural Radiance Field

In traditional 3D reconstruction using UAV images, only radiance information, which is treated as a geometric constraint, is used in feature matching, allowing for the restoration of the scene’s structure. After introducing radiance supervision, NeRF can adjust the geometry in the fixed-ray direction, resulting in a smaller search space and higher robustness. Considering the lack of NeRF construction methods for aerial scenarios, we propose a new NeRF point sampling method, which is generated using a UAV imaging model, compatible with a global geographic coordinate system, and suitable for a UAV view. We found that NeRF is optimized entirely based on the radiance while ignoring the direct geometry constraint. Therefore, we designed a radiance correction strategy that considers the incidence angle. Our method can complete point sampling in a UAV imaging scene, as well as simultaneously perform digital surface model construction and ground radiance information recovery. When tested on self-acquired datasets, the NeRF variant proposed in this paper achieved better reconstruction accuracy than the original NeRF-based methods. It also reached a level of precision comparable to that of traditional photogrammetry methods, and it is capable of outputting a surface albedo that includes shadow information.

𝒢‐Style: Stylized Gaussian Splatting

AbstractWe introduce 𝒢‐Style, a novel algorithm designed to transfer the style of an image onto a 3D scene represented using Gaussian Splatting. Gaussian Splatting is a powerful 3D representation for novel view synthesis, as—compared to other approaches based on Neural Radiance Fields—it provides fast scene renderings and user control over the scene. Recent pre‐prints have demonstrated that the style of Gaussian Splatting scenes can be modified using an image exemplar. However, since the scene geometry remains fixed during the stylization process, current solutions fall short of producing satisfactory results. Our algorithm aims to address these limitations by following a three‐step process: In a pre‐processing step, we remove undesirable Gaussians with large projection areas or highly elongated shapes. Subsequently, we combine several losses carefully designed to preserve different scales of the style in the image, while maintaining as much as possible the integrity of the original scene content. During the stylization process and following the original design of Gaussian Splatting, we split Gaussians where additional detail is necessary within our scene by tracking the gradient of the stylized color. Our experiments demonstrate that 𝒢‐Style generates high‐quality stylizations within just a few minutes, outperforming existing methods both qualitatively and quantitatively.

SCARF: Scalable Continual Learning Framework for Memory‐efficient Multiple Neural Radiance Fields

AbstractThis paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi‐layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo‐realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross‐scene weight matrix and a set of scene‐specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo‐realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state‐of‐the‐art rendering quality of continual learning NeRF on NeRF‐Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.

Cinematic Gaussians: Real‐Time HDR Radiance Fields with Depth of Field

AbstractRadiance field methods represent the state of the art in reconstructing complex scenes from multi‐view photos. However, these reconstructions often suffer from one or both of the following limitations: First, they typically represent scenes in low dynamic range (LDR), which restricts their use to evenly lit environments and hinders immersive viewing experiences. Secondly, their reliance on a pinhole camera model, assuming all scene elements are in focus in the input images, presents practical challenges and complicates refocusing during novel‐view synthesis. Addressing these limitations, we present a lightweight method based on 3D Gaussian Splatting that utilizes multi‐view LDR images of a scene with varying exposure times, apertures, and focus distances as input to reconstruct a high‐dynamic‐range (HDR) radiance field. By incorporating analytical convolutions of Gaussians based on a thin‐lens camera model as well as a tonemapping module, our reconstructions enable the rendering of HDR content with flexible refocusing capabilities. We demonstrate that our combined treatment of HDR and depth of field facilitates real‐time cinematic rendering, outperforming the state of the art.

True orthophoto generation from unmanned aerial vehicle imagery using neural radiance fields with multi-resolution hash encoding

True orthophoto generation from unmanned aerial vehicle imagery using neural radiance fields with multi-resolution hash encoding

GSEditPro: 3D Gaussian Splatting Editing with Attention‐based Progressive Localization

AbstractWith the emergence of large‐scale Text‐to‐Image(T2I) models and implicit 3D representations like Neural Radiance Fields (NeRF), many text‐driven generative editing methods based on NeRF have appeared. However, the implicit encoding of geometric and textural information poses challenges in accurately locating and controlling objects during editing. Recently, significant advancements have been made in the editing methods of 3D Gaussian Splatting, a real‐time rendering technology that relies on explicit representation. However, these methods still suffer from issues including inaccurate localization and limited manipulation over editing. To tackle these challenges, we propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. Leveraging the explicit nature of the 3D Gaussian distribution, we introduce an attention‐based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross‐attention layers of the T2I model. Furthermore, we present an innovative editing optimization method based on 3D Gaussian Splatting, obtaining stable and refined editing results through the guidance of Score Distillation Sampling and pseudo ground truth. We prove the efficacy of our method through extensive experiments.

Comparing the accuracy of 3D urban olive tree models detected by smartphone using LiDAR sensor, photogrammetry and NeRF: a case study of ’Ascolana Tenera’ in Italy

Abstract. Rapid urban growth makes green space management crucial to improve citizens’ well-being. Urban olive trees characterize the Italian landscapes and their culture. This study explores different methodologies for urban tree assessment in this context, using an iPhone 14 Pro Max. These included: 1) its integrated Light Detection and Ranging (LiDAR) sensor using the Recon3D app, 2) its camera with Structure from Motion (SfM) techniques, and 3) its camera for generating 3D models using Neural Radiance Fields (NeRF). Additionally, a professional Mobile Laser Scanner (MLS), was used for comparison. Total height (H), canopy base height (CBH) and canopy volume (CV) measurements were extracted using both CloudCompare and allometric formulas. The main aim of this paper is to compare the 3D models of olive trees obtained from low-cost sensors with those generated from the MLS, which is a more accurate device but comes with significantly higher costs. The results, in terms of RMSE (iPhone LiDAR - H: 0.46 m, CBH: 0.12 m, CV: 15.66 m3; iPhone-SfM - H: 0.95 m, CBH: 0.19 m, CV: 25.85 m3; iPhone-NeRF - H: 1.26 m, CBH: 0.31 m, CV: 33.79 m3), bias and volume differences, reveal that the smartphone, in all the methodologies, tends to underestimate measurements as the size of the trees increases. This is due to the higher MLS range of acquisition. Despite these limitations, low-cost solutions like smartphone-based methods can be a viable alternative given their economic accessibility.

VPRF: Visual Perceptual Radiance Fields for Foveated Image Synthesis.

Neural radiance fields (NeRF) has achieved revolutionary breakthrough in the novel view synthesis task for complex 3D scenes. However, this new paradigm struggles to meet the requirements for real-time rendering and high perceptual quality in virtual reality. In this paper, we propose VPRF, a novel visual perceptual based radiance fields representation method, which for the first time integrates the visual acuity and contrast sensitivity models of human visual system (HVS) into the radiance field rendering framework. Initially, we encode both the appearance and visual sensitivity information of the scene into our radiance field representation. Then, we propose a visual perceptual sampling strategy, allocating computational resources according to the HVS sensitivity of different regions. Finally, we propose a sampling weight-constrained training scheme to ensure the effectiveness of our sampling strategy and improve the representation of the radiance field based on the scene content. Experimental results demonstrate that our method renders more efficiently, with higher PSNR and SSIM in the foveal and salient regions compared to the state-of-the-art FoV-NeRF. The results of the user study confirm that our rendering results exhibit high-fidelity visual perception.

Scene-Constrained Neural Radiance Fields for High-Quality Sports Scene Rendering Based on Visual Sensor Network

Scene-Constrained Neural Radiance Fields for High-Quality Sports Scene Rendering Based on Visual Sensor Network

High-Fidelity and High-Efficiency Talking Portrait Synthesis With Detail-Aware Neural Radiance Fields.

In this paper, we propose a novel rendering framework based on neural radiance fields (NeRF) named HH-NeRF that can generate high-resolution audio-driven talking portrait videos with high fidelity and fast rendering. Specifically, our framework includes a detail-aware NeRF module and an efficient conditional super-resolution module. Firstly, a detail-aware NeRF is proposed to efficiently generate a high-fidelity low-resolution talking head, by using the encoded volume density estimation and audio-eye-aware color calculation. This module can capture natural eye blinks and high-frequency details, and maintain a similar rendering time as previous fast methods. Secondly, we present an efficient conditional super-resolution module on the dynamic scene to directly generate the high-resolution portrait with our low-resolution head. Incorporated with the prior information, such as depth map and audio features, our new proposed efficient conditional super resolution module can adopt a lightweight network to efficiently generate realistic and distinct high-resolution videos. Extensive experiments demonstrate that our method can generate more distinct and fidelity talking portraits on high resolution (900 × 900) videos compared to state-of-the-art methods. Our code is available at https://github.com/muyuWang/HHNeRF.

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

ABSTRACT Visual 3D reconstruction enables rebuilding 3D scenes from captured images, serving as a fundamental data source for digital earth modeling and intelligent cities. In the foundational step, recent methods leverage learning-based descriptors for image registration and achieve tremendous advances in precision and robustness. However, these methods inevitably execute down sampling towards high-resolution images to fit the needs of neural networks, which leads to precision degradation of feature localization and matching. Thus, we propose GeoRecon: a novel coarse-to-fine visual 3D reconstruction method that optimally utilizes high-resolution images for high-quality visual 3D reconstruction. Firstly, the coarse stage conducts coarse reconstruction from downsampled images by performing neural matching with geometric priors. Secondly, we define the fine-grained stage, proposing a GPU-based algorithm for generating image-patch correspondences based on the neural matching priors to perform fine-grained image registration. Finally, based on the optimized camera poses under this coarse-to-fine paradigm, progressive dense reconstruction leveraging efficient neural radiance fields is proposed to accomplish the high-quality MVS reconstruction. Comparative experiments across various scenarios demonstrate the proposed method’s superior precision, robustness, and reconstruction quality.

WaterHE-NeRF: Water-ray matching neural radiance fields for underwater scene reconstruction

Neural Radiance Field (NeRF) technology demonstrates immense potential in novel viewpoint synthesis tasks due to its physics-based volumetric rendering process, which is particularly promising in underwater scenes. However, existing underwater NeRF methods face challenges in handling light attenuation caused by the water medium and the lack of real Ground Truth (GT) supervision. To address these issues, we propose WaterHE-NeRF, a novel approach incorporating a water-ray matching field developed based on Retinex theory. This field precisely encodes color, density, and illuminance attenuation in three-dimensional space. WaterHE-NeRF employs an illuminance attenuation mechanism to generate degraded and clear multi-view images, optimizing image restoration by combining reconstruction loss with Wasserstein distance. Furthermore, using histogram equalization (HE) as pseudo-GT, WaterHE-NeRF enhances the network’s accuracy in preserving original details and color distribution. Extensive experiments on real underwater and synthetic datasets validate the effectiveness of WaterHE-NeRF.

Neighborhood transformer for sparse-view X-ray 3D foot reconstruction

In medical imaging, Sparse-View X-ray 3D reconstruction is crucial for analyzing and diagnosing foot bone structures. However, existing methods face limitations when handling sparse view data and complex bone structures. To enhance reconstruction accuracy and detail preservation, this paper proposes an innovative Sparse-View X-ray 3D foot reconstruction technique based on Neighborhood Transformer. A new Neighborhood Position Encoding strategy is introduced, which divides X-ray images into local regions using a window mechanism and precisely selects these regions through nearest neighbor methods, thereby capturing detailed features in the images. Building upon existing NeRF (Neural Radiance Fields) technology, the paper introduces the Neighborhood Transformer module. This module significantly improves the expression capability for complex foot bone structures through depthwise separable convolutions and a dual-branch local–global Transformer network. Additionally, an adaptive weight learning strategy is applied within the Transformer module, enabling the model to better capture long-distance dependencies, thereby improving its ability to handle sparse view data.

GS‐Octree: Octree‐based 3D Gaussian Splatting for Robust Object‐level 3D Reconstruction Under Strong Lighting

AbstractThe 3D Gaussian Splatting technique has significantly advanced the construction of radiance fields from multi‐view images, enabling real‐time rendering. While point‐based rasterization effectively reduces computational demands for rendering, it often struggles to accurately reconstruct the geometry of the target object, especially under strong lighting conditions. Strong lighting can cause significant color variations on the object's surface when viewed from different directions, complicating the reconstruction process. To address this challenge, we introduce an approach that combines octree‐based implicit surface representations with Gaussian Splatting. Initially, it reconstructs a signed distance field (SDF) and a radiance field through volume rendering, encoding them in a low‐resolution octree. This initial SDF represents the coarse geometry of the target object. Subsequently, it introduces 3D Gaussians as additional degrees of freedom, which are guided by the initial SDF. In the third stage, the optimized Gaussians enhance the accuracy of the SDF, enabling the recovery of finer geometric details compared to the initial SDF. Finally, the refined SDF is used to further optimize the 3D Gaussians via splatting, eliminating those that contribute little to the visual appearance. Experimental results show that our method, which leverages the distribution of 3D Gaussians with SDFs, reconstructs more accurate geometry, particularly in images with specular highlights caused by strong lighting. The source code can be downloaded from https://github.com/LaoChui999/GS-Octree.

Gaussian in the Dark: Real‐Time View Synthesis From Inconsistent Dark Images Using Gaussian Splatting

Abstract3D Gaussian Splatting has recently emerged as a powerful representation that can synthesize remarkable novel views using consistent multi‐view images as input. However, we notice that images captured in dark environments where the scenes are not fully illuminated can exhibit considerable brightness variations and multi‐view inconsistency, which poses great challenges to 3D Gaussian Splatting and severely degrades its performance. To tackle this problem, we propose Gaussian‐DK. Observing that inconsistencies are mainly caused by camera imaging, we represent a consistent radiance field of the physical world using a set of anisotropic 3D Gaussians, and design a camera response module to compensate for multi‐view inconsistencies. We also introduce a step‐based gradient scaling strategy to constrain Gaussians near the camera, which turn out to be floaters, from splitting and cloning. Experiments on our proposed benchmark dataset demonstrate that Gaussian‐DK produces high‐quality renderings without ghosting and floater artifacts and significantly outperforms existing methods. Furthermore, we can also synthesize light‐up images by controlling exposure levels that clearly show details in shadow areas.

CoupNeRF: Property‐aware Neural Radiance Fields for Multi‐Material Coupled Scenario Reconstruction

AbstractNeural Radiance Fields (NeRFs) have achieved significant recognition for their proficiency in scene reconstruction and rendering by utilizing neural networks to depict intricate volumetric environments. Despite considerable research dedicated to reconstructing physical scenes, rare works succeed in challenging scenarios involving dynamic, multi‐material objects. To alleviate, we introduce CoupNeRF, an efficient neural network architecture that is aware of multiple material properties. This architecture combines physically grounded continuum mechanics with NeRF, facilitating the identification of motion systems across a wide range of physical coupling scenarios. We first reconstruct specific‐material of objects within 3D physical fields to learn material parameters. Then, we develop a method to model the neighbouring particles, enhancing the learning process specifically in regions where material transitions occur. The effectiveness of CoupNeRF is demonstrated through extensive experiments, showcasing its proficiency in accurately coupling and identifying the behavior of complex physical scenes that span multiple physics domains.

Robust localization of shear connectors in accelerated bridge construction with neural radiance field

Accelerated bridge construction (ABC) demands precise alignment of prefabricated members to prevent assembly failure. Conventional methods struggle to localize shear connectors from point cloud data (PCD) generated by structure-from-motion due to its sparsity. This paper introduces a robust method for shear connector localization using PCD generated by a neural radiance field and a three-step narrowing-down algorithm. The PCD exhibits densely populated points for small connectors, allowing the algorithm to pinpoint their locations accurately. The method successfully identified all 72 shear connectors in a mock-up prefabricated girder, with an average error of 10 mm, demonstrating its potential for assessing constructability in ABC projects. Future research may integrate deep learning-based segmentation techniques to enhance efficiency and adaptability in complex geometries and non-standard bridge designs.