Large 3D Scenes Research Articles

BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

We present BlockFusion, a diffusion-based model that generates 3D scenes as unit blocks and seamlessly incorporates new blocks to extend the scene. BlockFusion is trained using datasets of 3D blocks that are randomly cropped from complete 3D scene meshes. Through per-block fitting, all training blocks are converted into the hybrid neural fields: with a tri-plane containing the geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the signed distance values. A variational auto-encoder is employed to compress the tri-planes into the latent tri-plane space, on which the denoising diffusion process is performed. Diffusion applied to the latent representations allows for high-quality and diverse 3D scene generation. To expand a scene during generation, one needs only to append empty blocks to overlap with the current scene and extrapolate existing latent tri-planes to populate new blocks. The extrapolation is done by conditioning the generation process with the feature samples from the overlapping tri-planes during the denoising iterations. Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that harmoniously blend with the existing scene. A 2D layout conditioning mechanism is used to control the placement and arrangement of scene elements. Experimental results indicate that BlockFusion is capable of generating diverse, geometrically consistent and unbounded large 3D scenes with unprecedented high-quality shapes in both indoor and outdoor scenarios.

AFRNet: Anchor-Free Object Detection Using Roadside LiDAR in Urban Scenes

In urban settings, roadside infrastructure LiDAR is a ground-based remote sensing system that collects 3D sparse point clouds for the traffic object detection of vehicles, pedestrians, and cyclists. Current anchor-free algorithms for 3D point cloud object detection based on roadside infrastructure face challenges related to inadequate feature extraction, disregard for spatial information in large 3D scenes, and inaccurate object detection. In this study, we propose AFRNet, a two-stage anchor-free detection network, to address the aforementioned challenges. We propose a 3D feature extraction backbone based on the large sparse kernel convolution (LSKC) feature set abstraction module, and incorporate the CBAM attention mechanism to enhance the large scene feature extraction capability and the representation of the point cloud features, enabling the network to prioritize the object of interest. After completing the first stage of center-based prediction, we propose a refinement method based on attentional feature fusion, where fused features incorporating raw point cloud features, voxel features, BEV features, and key point features are used for the second stage of refinement to complete the detection of 3D objects. To evaluate the performance of our detection algorithms, we conducted experiments using roadside LiDAR data from the urban traffic dataset DAIR-V2X, based on the Beijing High-Level Automated Driving Demonstration Area. The experimental results show that AFRNet has an average of 5.27 percent higher detection accuracy than CenterPoint for traffic objects. Comparative tests further confirm that our method achieves high accuracy in roadside LiDAR object detection.

Investigate methods for smooth transitions between levels of detail for effective visualization of 3D spaces

The object of study is 3D-rendering. The subject of study is providing smooth transitions between LOD models. The purpose of the work is to increase rendering performance of large 3D-scenes considering smooth transitions between models with different levels of detail. The study examines existing algorithms for enabling smooth transitions between levels of detail considering computational complexity and visual appeal of said methods.

SMS-Net: Sparse multi-scale voxel feature aggregation network for LiDAR-based 3D object detection

Real-time three-dimensional (3D) object detection has become a crucial component of autonomous driving applications. Recent research demonstrates that a voxel-based feature aggregation method is accurate and efficient in large 3D scenes. However, the parameter choice of voxel size has become a sensitive issue because of the contradiction between its detection performance and inference speed. To alleviate this problem, in this paper we propose a sparse multi-scale voxel feature aggregation network (SMS-Net), a novel one-stage, end-to-end network that primarily contains a sparse multi-scale-fusion (SMSF) module and shallow-to-deep regression (SDR) module. First, the raw point clouds are divided into different scales of voxels to construct diverse 3D sparse feature maps. Then, the SMSF module attentively aggregates the point-wise features with a perspective-channel attention mechanism and fuses multi-scale features at 3D sparse feature-map level to achieve more fine-grained shape information. In addition, the new SDR module boosts the localization accuracy and 3D box estimation accuracy through multiple aggregation at feature-map level, which requires less computational overhead. Extensive experiments demonstrate the remarkable performance improvements from each module of the proposed method. On the KITTI 3D object detection benchmark, for example, SMS-Net outperforms most one-stage, state-of-the-art methods and its performance can even be compared to that of two-stage methods. These detection results are achieved with a real-time inference speed of 42 Hz. SMS-Net also achieves state-of-the-art performance on the nuScenes 3D benchmark.

Efficient Ray Tracing of Large 3D Scenes for Mobile Distributed Computing Environments.

Cluster computing has attracted much attention as an effective way of solving large-scale problems. However, only a few attempts have been made to explore mobile computing clusters that can be easily built using commodity smartphones and tablets. To investigate the possibility of mobile cluster-based rendering of large datasets, we developed a mobile GPU ray tracer that renders nontrivial 3D scenes with many millions of triangles at an interactive frame rate on a small-scale mobile cluster. To cope with the limited processing power and memory space, we first present an effective 3D scene representation scheme suitable for mobile GPU rendering. Then, to avoid performance impairment caused by the high latency and low bandwidth of mobile networks, we propose using a static load balancing strategy, which we found to be more appropriate for the vulnerable mobile clustering environment than a dynamic strategy. Our mobile distributed rendering system achieved a few frames per second when ray tracing 1024 × 1024 images, using only 16 low-end smartphones, for large 3D scenes, some with more than 10 million triangles. Through a conceptual demonstration, we also show that the presented rendering scheme can be effectively explored for augmenting real scene images, captured or perceived by augmented and mixed reality devices, with high quality ray-traced images.

On the role of geometry in geo-localization

Consider the geo-localization task of finding the pose of a camera in a large 3D scene from a single image. Most existing CNN-based methods use as input textured images. We aim to experimentally explore whether texture and correlation between nearby images are necessary in a CNN-based solution for the geo-localization task. To do so, we consider lean images, textureless projections of a simple 3D model of a city. They only contain information related to the geometry of the scene viewed (edges, faces, and relative depth). The main contributions of this paper are: (i) to demonstrate the ability of CNNs to recover camera pose using lean images; and (ii) to provide insight into the role of geometry in the CNN learning process.

Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network.

We have developed a series of 3D endoscopic systems where a micro-sized pattern projector is inserted through the instrument channel of the endoscope and shapes are reconstructed by a structured light technique using captured images of the endoscopic camera. One problem of the previous works is that the accuracy of shape reconstruction is low, because the projector cannot be fixed to the endoscope, and thus, the pose of the pattern projector w.r.t. the camera cannot be pre-calibrated. In this paper, we propose a method to auto-calibrate the pose of the projector without using any special devices nor manual process. Since the technique is one-shot, multiple shapes can be reconstructed from an image sequence and a large 3D scene can be recovered by merging them. Experiments are conducted using the real system.

A 3D roaming and collision detection algorithm applicable for massive spatial data.

In this paper, a novel 3D roaming algorithm considering collision detection and interaction is proposed that adopts a triangle mesh to organize and manage massive spatial data and uses a customized bounding box intersector to rapidly obtain the potential collided triangles. The proposed algorithm can satisfy the requirements of timeliness and practicability during complicated large 3D scene collision detection. Moreover, we designed a method to calculate the collision point coordinates according to the spatial position relation and distance change between the virtual collision detection sphere and triangles, with the triangle edges and three vertices being considered. Compared to the methods that use the native intersector of OpenSceneGraph (OSG) to obtain the collision point coordinates, the calculation efficiency of the proposed method is greatly improved. Usually, when there is a big split/pit in the scene, the viewpoints will fly off the scene due to the fall of the collision detection sphere, or the region interior cannot be accessed when the entrance of some local region (e.g., internal grotto) of the scene is too small. These problems are solved in this paper through 3D scene-path training and by self-adaptively adjusting the radius of the virtual collision detection sphere. The proposed 3D roaming and collision detection method applicable for massive spatial data overcomes the limitation that the existing roaming and collision detection methods are only applicable to 3D scenes with a small amount of data and simple models. It provides technical supports for freewill browsing and roaming of indoor/outdoor and overground/underground of the 3D scene in cases of massive spatial data.

Multi-Window 3D Interaction for Collaborative Virtual Reality.

We present a novel collaborative virtual reality system that offers multiple immersive 3D views at large 3D scenes. The physical setup consists of two synchronized multi-user 3D displays: a tabletop and a large vertical projection screen. These displays afford different presentations of the shared 3D scene. The wall display lends itself to the egocentric exploration at 1:1 scale, while the tabletop affords an allocentric overview. Additionally, handheld 3D portals facilitate the personal exploration of the scene, the comparison of views, and the exchange with others. Our developments enable seamless 3D interaction across these independent 3D views. This requires the simultaneous representation of user input in the different viewing contexts. However, the resulting interactions cannot be executed independently. The application must coordinate the interactions and resolve potential ambiguities to provide plausible effects. We analyze and document the challenges of seamless 3D interaction across multiple independent viewing windows, propose a high-level software design to realize the necessary functionality, and apply the design to a set of interaction tools. Our setup was tested in a formal user study, which revealed general advantages of collaborative 3D data exploration with multiple views in terms of user preference, comfort, and task performance.

Language-driven synthesis of 3D scenes from scene databases

We introduce a novel framework for using natural language to generate and edit 3D indoor scenes, harnessing scene semantics and text-scene grounding knowledge learned from large annotated 3D scene databases. The advantage of natural language editing interfaces is strongest when performing semantic operations at the sub-scene level, acting on groups of objects. We learn how to manipulate these sub-scenes by analyzing existing 3D scenes. We perform edits by first parsing a natural language command from the user and transforming it into a semantic scene graph that is used to retrieve corresponding sub-scenes from the databases that match the command. We then augment this retrieved sub-scene by incorporating other objects that may be implied by the scene context. Finally, a new 3D scene is synthesized by aligning the augmented sub-scene with the user's current scene, where new objects are spliced into the environment, possibly triggering appropriate adjustments to the existing scene arrangement. A suggestive modeling interface with multiple interpretations of user commands is used to alleviate ambiguities in natural language. We conduct studies comparing our approach against both prior text-to-scene work and artist-made scenes and find that our method significantly outperforms prior work and is comparable to handmade scenes even when complex and varied natural sentences are used.

Compact large size colour 3D dynamic holographic display using liquid crystal display panel

Large three-dimensional (3D) scene reconstruction is a challenge for current colour holographic displays. In this study, we develop a compact display system to reconstruct a large 3D scene using a 21-inch liquid crystal display (LCD) panel that serves as a spatial light modulator (SLM). We utilise a Fresnel lens instead of a conventional lens to reduce the complexity of the system. A spatial multiplexing colour filter attached to the SLM is used to simplify the structure of the colour display. Additionally, an independent block holographic algorithm is developed to record accurate depth information. Numerical simulations and optical experiments are performed to verify the proposed system. The optically reconstructed 3D colour scene is 480 mm × 300 mm × 500 mm in size. Therefore, the proposed display system represents development toward a commercial product for its low device requirements.

Fast accessing Web3D contents using lightweight progressive meshes

AbstractAccessing Web3D contents is relatively slow through Internet under limited bandwidth. Preprocessing of 3D models can certainly alleviate the problem, such as 3D compression and progressive meshes (PM). But none of them considers the similarity between components of a 3D model, so that we could take advantage of this to further improve the efficiency. This paper proposes a similarity‐aware data reduction method together with PM, called lightweight progressive meshes (LPM). LPM aims to excavate similar components in a 3D model, generates PM representation of each component left after removing redundant components, and organizes all the processed data using a structure called lightweight scene graph. The proposed LPM possesses four significant advantages. First, it can minimize the file size of 3D model dramatically without almost any precision loss. Because of this, minimal data is delivered. Second, PM enables the delivery to be progressive, so called streaming. Third, when rendering at client side, due to lightweight scene graph, decompression is not necessary and instanced rendering is fully exerted. Fourth, it is extremely efficient and effective under very limited bandwidth, especially when delivering large 3D scenes. Performance on real data justifies the effectiveness of our LPM, which improves the state‐of‐the‐art in accessing Web3D contents. Copyright © 2015 John Wiley & Sons, Ltd.

Adaptive modeling and distribution of large natural scenes

This thesis deals with the modeling and the interactive streaming of large natural 3D scenes. We aim at providing techniques to allow the remote walkthrough of users in a natural 3D scene ensuring botanical coherency and interactivity.

Graph cut based panoramic 3D modeling and ground truth comparison with a mobile platform – The Wägele

Efficient and comfortable acquisition of large 3D scenes is an important topic for many current and future applications in the field of robotics, factory and office visualization, 3DTV and cultural heritage. In this paper we present both an omnidirectional stereo vision approach for 3D modeling based on graph cut techniques and also a new mobile 3D model acquisition platform where it is employed. The platform comprises a panoramic camera and a 2D laser range scanner for self localization by scan matching. 3D models are acquired just by moving the platform around and recording images in regular intervals. Additionally, we concurrently build 3D models using two supplementary laser range scanners. This enables the investigation of the stereo algorithm’s quality by comparing it with the laser scanner based 3D model as ground truth. This offers a more objective point of view on the achieved 3D model quality.

3D Large-Scale Seabed Reconstruction for UUV Simultaneous Localization and Mapping

This paper proposes a new technique to reconstruct large 3D scenes from a sequence of video images by combining Bayesian filtering and state-of-art 3D computer vision. The approach performs the alignment of a sequence of 3D partial reconstructions of the seafloor thanks to the re-observations of passive landmarks by means of a linear Kalman filter-based SLAM approach. Landmarks are detected on the images and characterized considering 2D and 3D features. Landmarks are re-observed while the robot is navigating and data association becomes easier but robust. Preliminary results are performed in virtual scenarios but processing real images synthetized from underwater textures.

A finite element–spherical harmonics model for radiative transfer in inhomogeneous clouds: Part I. The EVENT model

We present a new numerical radiative transfer model for application to solar radiation transport in three-dimensional (3D) cloudy atmospheres. The code uses the finite element–spherical harmonics (FE– P N ) approximation to solve the second-order even-parity form of the transport equation. It is validated by comparison with solutions from two well-established, deterministic radiative transfer models, the one-dimensional (1D) DISORT code and the spherical harmonics discrete ordinates method (SHDOM). The cases solved show generally good agreement, but also reveal some differences. EVEn-parity Neutral particle Transport (EVENT) is very efficient at performing 1D calculations quickly and, even for quite high angular resolutions, is faster. EVENT also has a competitive speed for the simpler, less-heterogeneous multidimensional cases but it is slower than SHDOM for more variable cases. However, there is significant potential to improve the performance of EVENT; it has not yet been optimised for speed and, as such, is not a finished product. Even as it is, EVENT could be used to produce fast, lower-resolution estimates for applications where this would be sufficient, or at lower-spatial resolutions with partial homogenisation. Another difference between the models is that the SHDOM algorithm is designed for small-scale inhomogeneous cloud fields in which the grid spacing is comparable to the mean free path. Problems arise from the use of larger grid cell optical depths, with an increase in the number of iterations required and a lack of flux conservation. Neither of the models is specifically constrained to conserve flux, but the conservation of flux gives an indication of the accuracy of the solution. Increasing the spatial and angular resolution can improve the accuracy but this is not always possible for very large 3D scenes. The grid-point method of property definition in SHDOM means that it performs best for cases with a continuous variation in extinction as this avoids discontinuities in the source function. The finite clouds used in our tests have sharp boundaries that are easily defined in EVENT but the difficulties caused by these in SHDOM are evident in excesses of up to 10 fluxes. The EVENT mesh resolution is determined by the local optical depth; it has no problem in dense areas but has more trouble in coping with voids or optically thin regions. These conditions are easily handled in SHDOM through streaming of photons along discrete ordinates but EVENT must implement methods such as a ray-tracing algorithm. Cases with extreme values of the extinction coefficient are probably best avoided with EVENT, but slightly larger-scale cases with greater optical depths, not suitable for SHDOM, may be solved more easily. These factors should be considered when selecting the most appropriate method for a particular application.

Efficient representation and streaming of 3D scenes

In the last few years several approaches (Proceedings of the Second International Workshop on Distributed Interactive Simulation and Real-Time Application, 1998, pp. 88–91; EUROGRAPHICS 2001, vol. 20(3) 2001) have been presented, which address the transmission and visualization of 3D scenes on distributed devices with different capabilities. These techniques are in the interest of a wide field of applications such as virtual chatrooms, product presentations, CAVE visualizations, 3D simulations, or 3D online games. In order to adapt the immense data effort to the capabilities of the devices and the networks, two basic problems have to be solved: the selection of the information according to the user's interest and the reduction of the selected data. Usually the first problem is reduced to visual aspects, which can be determined by visibility culling algorithms. This paper concentrates on the second problem and introduces a system in order to stream the data of even large 3D scenes to remote devices.