3D Scene Research Articles

A 3D interactive scene construction method for interior design based on virtual reality

A 3D interactive scene construction method for interior design based on virtual reality

Robotic Grasping Detection Algorithm Based on 3D Vision Dual-Stream Encoding Strategy

The automatic generation of stable robotic grasping postures is crucial for the application of computer vision algorithms in real-world settings. This task becomes especially challenging in complex environments, where accurately identifying the geometric shapes and spatial relationships between objects is essential. To enhance the capture of object pose information in 3D visual scenes, we propose a planar robotic grasping detection algorithm named SU-Grasp, which simultaneously focuses on local regions and long-distance relationships. Built upon a U-shaped network, SU-Grasp introduces a novel dual-stream encoding strategy using the Swin Transformer combined with spatial semantic enhancement. Compared to existing baseline methods, our algorithm achieves superior performance across public datasets, simulation tests, and real-world scenarios, highlighting its robust understanding of complex spatial environments.

Deep Learning-Driven Virtual Furniture Replacement Using GANs and Spatial Transformer Networks

This study proposes a Generative Adversarial Network (GAN)-based method for virtual furniture replacement within indoor scenes. The proposed method addresses the challenge of accurately positioning new furniture in an indoor space by combining image reconstruction with geometric matching through combining spatial transformer networks and GANs. The system leverages deep learning architectures like Mask R-CNN for executing image segmentation and generating masks, and it employs DeepLabv3+, EdgeConnect algorithms, and ST-GAN networks for carrying out virtual furniture replacement. With the proposed system, furniture shoppers can obtain a virtual shopping experience, providing an easier way to understand the aesthetic effects of furniture rearrangement without putting in effort to physically move furniture. The proposed system has practical applications in the furnishing industry and interior design practices, providing a cost-effective and efficient alternative to physical furniture replacement. The results indicate that the proposed method achieves accurate positioning of new furniture in indoor scenes with minimal distortion or displacement. The proposed system is limited to 2D front-view images of furniture and indoor scenes. Future work would involve synthesizing 3D scenes and expanding the system to replace furniture images photographed from different angles. This would enhance the efficiency and practicality of the proposed system for virtual furniture replacement in indoor scenes.

Enhanced TomoSAR imaging method with complex least squares for 3D building reconstruction

ABSTRACT Synthetic aperture radar tomography (TomoSAR) has emerged as an advanced technology for the rapid acquisition of three-dimensional information, with model solutions dependent on a specific complex least squares adjustment criterion. We design two TomoSAR imaging algorithms based on two adjustment criterions minimizing the square sum of the real and imaginary components of observation vector residual, and minimizing the square sum of the L2-norm of observation vector residual. Additionally, we propose a strong scatterer recognition method that combines amplitude thresholding and density-based spatial clustering of applications with noise to alleviate the low efficiency and excessive noise in real images. The simulated and practical experiments are conducted to evaluate the performance of algorithms. Results suggested that two algorithms perform similarly in parameter estimation accuracy and noise immunity for buildings with simple structures and single scatterer. However, for buildings with multiple scatterers, the algorithm derived by minimizing the square sum of the L2-norm from the observation vector residual exhibits more robustly than the other algorithm, accurately distinguishing multiple scatterers within pixels and reconstructing 3D scenes mirroring the actual ones. The results verify the effectiveness of the proposed strong scatterer recognition method and the importance of the adjustment criterion in TomoSAR imaging.

𝒢‐Style: Stylized Gaussian Splatting

AbstractWe introduce 𝒢‐Style, a novel algorithm designed to transfer the style of an image onto a 3D scene represented using Gaussian Splatting. Gaussian Splatting is a powerful 3D representation for novel view synthesis, as—compared to other approaches based on Neural Radiance Fields—it provides fast scene renderings and user control over the scene. Recent pre‐prints have demonstrated that the style of Gaussian Splatting scenes can be modified using an image exemplar. However, since the scene geometry remains fixed during the stylization process, current solutions fall short of producing satisfactory results. Our algorithm aims to address these limitations by following a three‐step process: In a pre‐processing step, we remove undesirable Gaussians with large projection areas or highly elongated shapes. Subsequently, we combine several losses carefully designed to preserve different scales of the style in the image, while maintaining as much as possible the integrity of the original scene content. During the stylization process and following the original design of Gaussian Splatting, we split Gaussians where additional detail is necessary within our scene by tracking the gradient of the stylized color. Our experiments demonstrate that 𝒢‐Style generates high‐quality stylizations within just a few minutes, outperforming existing methods both qualitatively and quantitatively.

SCARF: Scalable Continual Learning Framework for Memory‐efficient Multiple Neural Radiance Fields

AbstractThis paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi‐layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo‐realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross‐scene weight matrix and a set of scene‐specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo‐realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state‐of‐the‐art rendering quality of continual learning NeRF on NeRF‐Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.

GSEditPro: 3D Gaussian Splatting Editing with Attention‐based Progressive Localization

AbstractWith the emergence of large‐scale Text‐to‐Image(T2I) models and implicit 3D representations like Neural Radiance Fields (NeRF), many text‐driven generative editing methods based on NeRF have appeared. However, the implicit encoding of geometric and textural information poses challenges in accurately locating and controlling objects during editing. Recently, significant advancements have been made in the editing methods of 3D Gaussian Splatting, a real‐time rendering technology that relies on explicit representation. However, these methods still suffer from issues including inaccurate localization and limited manipulation over editing. To tackle these challenges, we propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. Leveraging the explicit nature of the 3D Gaussian distribution, we introduce an attention‐based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross‐attention layers of the T2I model. Furthermore, we present an innovative editing optimization method based on 3D Gaussian Splatting, obtaining stable and refined editing results through the guidance of Score Distillation Sampling and pseudo ground truth. We prove the efficacy of our method through extensive experiments.

Generating high-quality phase-only holograms of binary images using global loss and stochastic homogenization training strategy

Deep learning presents a highly effective for computer-generated holography (CGH). However, existing learning-based CGH algorithms may encounter significant obstacles in generating high-quality holograms for binary images. This study proposes a novel diffraction model-driven neural network, termed load sharing yielding holography (LSY-Holo), which leverages a dual-branch architecture and a global structural similarity index measure (Global_SSIM) loss function to generate phase-only holograms (POHs). LSY-Holo framework facilitates capturing global information, enhancing the network’s performance to address the ill-posed inverse problem. To further optimize the model for binary images, the research introduces a custom Binary dataset alongside a stochastic homogenization preprocessing method (SHPM), which mitigates the prevalent issue of excessive noise in reconstructed images. Additionally, this deep learning approach is integrated with an iterative algorithm, which employs holograms rapidly generated by LSY-Holo as the initial phase of the AdamW Holography (AH) algorithm. This strategy significantly accelerates the convergence of the iterative process. LSY-Holo is further extended to 3D scenes, enabling dynamic defocusing and focusing effects at varying depths. Results from numerical simulations and full-color optical reconstruction experiments demonstrate that the proposed method can enhance image quality and effectively reduce the artifacts in reconstructed images.

Digital-twin-based multi-scale simulation supports urban emergency management: a case study of urban epidemic transmission

ABSTRACT Improving emergency management capabilities is vital for sustainable cities. However, urban emergency management is a complex, giant system that interacts between urban virtual and real spaces, requiring simulation in the virtual space and feedback to the real space. The digital twin model can potentially play an important role in this process, but still faces significant challenges in terms of modeling, simulation, and visualization. In this study, we propose a digital-twin-based multi-scale simulation method and take epidemic transmission as a case study. First, an entity-oriented unified urban database integrating multi-source urban data was constructed as the digital twins of urban human settlements for emergency modeling. Then, a multi-scale urban emergency simulation method in 3D geographic environment was proposed to support emergency simulation for multi-level urban management units. Moreover, an adaptive fusion rendering of emergency information and 3D scene was proposed to support the refinement of emergency visualization. An experimental case study on epidemic risk estimation was conducted through thousands of outbreak scenarios. The results demonstrate that the proposed method can establish the digital twins of urban human settlements, where modeling, simulation, and visualization of emergency at multi-scales can be carried out, thus enhancing the efficacy and efficiency of emergency management.

VPRF: Visual Perceptual Radiance Fields for Foveated Image Synthesis.

Neural radiance fields (NeRF) has achieved revolutionary breakthrough in the novel view synthesis task for complex 3D scenes. However, this new paradigm struggles to meet the requirements for real-time rendering and high perceptual quality in virtual reality. In this paper, we propose VPRF, a novel visual perceptual based radiance fields representation method, which for the first time integrates the visual acuity and contrast sensitivity models of human visual system (HVS) into the radiance field rendering framework. Initially, we encode both the appearance and visual sensitivity information of the scene into our radiance field representation. Then, we propose a visual perceptual sampling strategy, allocating computational resources according to the HVS sensitivity of different regions. Finally, we propose a sampling weight-constrained training scheme to ensure the effectiveness of our sampling strategy and improve the representation of the radiance field based on the scene content. Experimental results demonstrate that our method renders more efficiently, with higher PSNR and SSIM in the foveal and salient regions compared to the state-of-the-art FoV-NeRF. The results of the user study confirm that our rendering results exhibit high-fidelity visual perception.

A real scene 3D Model-Driven sunlight analysis method for complex building roofs

A real scene 3D Model-Driven sunlight analysis method for complex building roofs

Inhibition of return in a 3D scene depends on the direction of depth switch between cue and target.

Inhibition of return (IOR) is a phenomenon that reflects slower target detection when the target appears at a previously cued rather than uncued location. In the present study, we investigated the extent to which IOR occurs in three-dimensional (3D) scenes comprising pictorial depth information. Peripheral cues and targets appeared on top of 3D rectangular boxes placed on the surface of a textured ground plane in virtual space. When the target appeared at a farther location than the cue, the magnitude of the IOR effect in the 3D condition remained similar to the values found in the two-dimensional (2D) control condition (IOR was depth-blind). When the target appeared at a nearer location than the cue, the magnitude of the IOR effect was significantly attenuated (IOR was depth-specific). The present findings address inconsistencies in the literature on the effect of depth on IOR and support the notion that visuospatial attention exhibits a near-space advantage even in 3D scenes consisting entirely of pictorial depth information.

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

ABSTRACT Visual 3D reconstruction enables rebuilding 3D scenes from captured images, serving as a fundamental data source for digital earth modeling and intelligent cities. In the foundational step, recent methods leverage learning-based descriptors for image registration and achieve tremendous advances in precision and robustness. However, these methods inevitably execute down sampling towards high-resolution images to fit the needs of neural networks, which leads to precision degradation of feature localization and matching. Thus, we propose GeoRecon: a novel coarse-to-fine visual 3D reconstruction method that optimally utilizes high-resolution images for high-quality visual 3D reconstruction. Firstly, the coarse stage conducts coarse reconstruction from downsampled images by performing neural matching with geometric priors. Secondly, we define the fine-grained stage, proposing a GPU-based algorithm for generating image-patch correspondences based on the neural matching priors to perform fine-grained image registration. Finally, based on the optimized camera poses under this coarse-to-fine paradigm, progressive dense reconstruction leveraging efficient neural radiance fields is proposed to accomplish the high-quality MVS reconstruction. Comparative experiments across various scenarios demonstrate the proposed method’s superior precision, robustness, and reconstruction quality.

3D reconstruction from focus for lensless imaging

The lensless camera is an ultra-thin imaging system that utilizes encoding elements instead of lenses to perceive the light field and reconstruct it through computational methods. Early studies have demonstrated that lensless cameras can encode 3D scenes at various depths in caustic patterns with varying sizes, known as point spread functions (PSFs). By deconvolving measurements with these PSFs, the reconstruction exhibits distinct focusing effects: objects in the focal plane appear sharp, while objects in other planes become blurred. Building upon this feature, we propose a feedforward network based on depth from focus to generate the depth map and the all-in-focus image by reconstructing the focal stack and deriving the probability of pixel clarity. Using our optimization framework, we present superior and more stable depth estimation than previous methods in both simulated data and real measurements captured by our lensless camera.

AEFF-SSC: an attention-enhanced feature fusion for 3D semantic scene completion

Abstract Three-dimensional (3D) occupancy perception technology aims to enable autonomous vehicles to observe and understand dense 3D environments. Estimating the complete geometry and semantics of a scene solely from visual images is challenging. However, humans can easily conceive the complete form of objects based on partial key information and their own experience. This ability is crucial for recognizing and interpreting the surrounding environment. To equip 3D occupancy perception systems with a similar capability, a 3D semantic scene completion method called AEFF-SSC is proposed. This method deeply explores boundary and multi-scale information in voxels, aiming to reconstruct 3D geometry more accurately. We have specifically designed an attention-enhanced feature fusion module that effectively fuses image feature information from different scales and focuses on feature boundary information, thereby more efficiently extracting voxel features. Additionally, we introduce a semantic segmentation module driven by a 3D attention-UNet network. This module combines a 3D U-Net network with a 3D attention mechanism. Through feature fusion and feature weighting, it aids in restoring 3D spatial information and significantly improves the accuracy of segmentation results. Experimental verification on the SemanticKITTI dataset demonstrates that AEFF-SSC significantly outperforms other existing methods in terms of both geometry and semantics. Specifically, within a 12.8 m × 12.8 m area ahead, our geometric occupancy accuracy has achieved a significant improvement of 71.58%, and at the same time, the semantic segmentation accuracy has also increased remarkably by 54.20%.

DSEM-NeRF: Multimodal feature fusion and global–local attention for enhanced 3D scene reconstruction

3D scene understanding often faces the problems of insufficient detail capture and poor adaptability to multi-view changes. To this end, we proposed a NeRF-based 3D scene understanding model DSEM-NeRF, which effectively improves the reconstruction quality of complex scenes through multimodal feature fusion and global–local attention mechanism. DSEM-NeRF extracts multimodal features such as color, depth, and semantics from multi-view 2D images, and accurately captures key areas by dynamically adjusting the importance of features. Experimental results show that DSEM-NeRF outperforms many existing models on the LLFF and DTU datasets, with PSNR reaching 20.01, 23.56, and 24.58 respectively, and SSIM reaching 0.834. In particular, it shows strong robustness in complex scenes and multi-view changes, verifying the effectiveness and reliability of the model.

New Solution to 3D Projection in Human-like Binocular Vision

A human eye has about 120 million rod cells and 6 million cone cells. This huge number of light-sensing cells inside a human eye will continuously produce a huge quantity of visual signals that flow into the human brain for daily processing. However, the real-time processing of these visual signals does not cause excessive energy consumption by the human brain. This fact tells us the truth which is to say that human-like vision processes do not rely on complicated and expensive formulas to compute depth, displacement, and colors. On the other hand, the human eye is like a camera with pan-tilt (PT) motions. We all know that in computer vision, each set of PT parameters (i.e., coefficients of pan motion as well as tilt motion) requires a dedicated calibration to determine a camera’s projection matrix. Since there is an infinite number of PT parameters that could be produced by a human eye, it is unlikely that a human brain stores an infinite number of calibration matrices for each human eye. These observations inspire us to look for a simpler and computationally non-expensive solution which is to undertake three-dimensional (3D) projection in human-like binocular vision. In other words, it is an interesting question for us to answer, which is to say whether simpler and learning-friendly formulas for computing depth and displacement exist or not. If the answer is yes, these formulas must also be calibration-friendly (i.e., easy process on the fly or on the go). In this paper, we present an important discovery of a new solution to 3D projection in a human-like binocular vision system. This solution is computationally simpler and could be easily learned or calibrated on the fly. We know that the purpose of doing 3D projection in binocular vision is to undertake forward and inverse transformations (or mappings) between coordinates in 2D digital images and coordinates in a 3D analogue scene. The formulas underlying the new solution are accurate, easily computable, easily tunable (i.e., to be calibrated on the fly or on the go) and could be easily implemented by a neural system (i.e., a network of neurons or a network of computational flows). Experimental results have validated the newly derived formulas which are better than textbook solutions.

High-Precision Disparity Estimation for Lunar Scene Using Optimized Census Transform and Superpixel Refinement

High-precision lunar scene 3D data are essential for lunar exploration and the construction of scientific research stations. Currently, most existing data from orbital imagery offers resolutions up to 0.5–2 m, which is inadequate for tasks requiring centimeter-level precision. To overcome this, our research focuses on using in situ stereo vision systems for finer 3D reconstructions directly from the lunar surface. However, the scarcity and homogeneity of available lunar surface stereo datasets, combined with the Moon’s unique conditions—such as variable lighting from low albedo, sparse surface textures, and extensive shadow occlusions—pose significant challenges to the effectiveness of traditional stereo matching techniques. To address the dataset gap, we propose a method using Unreal Engine 4 (UE4) for high-fidelity physical simulation of lunar surface scenes, generating high-resolution images under realistic and challenging conditions. Additionally, we propose an optimized cost calculation method based on Census transform and color intensity fusion, along with a multi-level super-pixel disparity optimization, to improve matching accuracy under harsh lunar conditions. Experimental results demonstrate that the proposed method exhibits exceptional robustness and accuracy in our soon-to-be-released multi-scene lunar dataset, effectively addressing issues related to special lighting conditions, weak textures, and shadow occlusion, ultimately enhancing disparity estimation accuracy.

Research on Spatial Data Aggregation Based on Aggregation Degree Function

Abstract. Restricted by the development level of hardware and software technology, computers cannot load massive data at one time, coupled with the fact that the three-dimensional linear scene is characterized by complicated lines, irregular shapes of features and a large number of features. Therefore, it is necessary to seek a three-dimensional spatial data organization and scheduling method applicable to the railroad scene to realize the display and interaction of the scene. In this paper, we propose a data organization method based on the aggregation degree function for the 3D model of features. The feature model is projected to generate an orthographic projection map, the cluster division rule is designed to divide the subsets, the feature model after clustering is aggregated using the feature model aggregation rule, and the aggregated model is displayed according to the aggregation degree function and viewpoint factor. The results of the test applied to the railroad scene show that the 3D spatial data aggregation based on the aggregation degree function can improve the loading speed of the 3D scene, and can effectively support the application of interactive visualization of the railroad 3D scene.

Rapid Development of A Spatial Digital Twins Using Open-source Solutions

Abstract. City-model spatial Digital Twins are a three-dimensional virtual self-updating twin of a city which is updated by real-time data. This paper explores the effectiveness of open-source data and tools that can generate a Digital Twin to be hosted on a distributed system. A spatial Digital Twin was successfully generated from a city-model made from open-source spatial data with access to real-time data. The Digital Twin application was hosted on a three-tier systems architecture. The exception to the open-source data was a Digital Surface Model (DSM) obtained from a private source. However, the DSM was an optional component. The output only met the minimum requirements of a Digital Twin while the visualization was basic where the live data was limited to global weather data rather than city-specific data. The investigation indicated that even with little data, generating a 3D city scene with an automated live data flow meets the spatial Digital Twin’s minimum requirements according to Piroumian (2023). The main contribution of the study is highlighting methods for generating and hosting a spatial Digital Twin and its associated data onto the web through using a three-tier architecture to store, host and display the Twin onto the web. The areas identified for further research include investigating variations in the distributed systems for hosting and publishing a spatial Digital Twin onto the web. A secondary subject of investigation is exploring methodologies for improving the detail of the spatial Digital Twin’s data for areas that do not receive much live data.