3D Scene Research Articles

Divide and Conquer: Improving Multi-Camera 3D Perception With 2D Semantic-Depth Priors and Input-Dependent Queries.

3D perception tasks, such as 3D object detection and Bird's-Eye-View (BEV) segmentation using multi-camera images, have drawn significant attention recently. Despite the fact that accurately estimating both semantic and 3D scene layouts are crucial for this task, existing techniques often neglect the synergistic effects of semantic and depth cues, leading to the occurrence of classification and position estimation errors. Additionally, the input-independent nature of initial queries also limits the learning capacity of Transformer-based models. To tackle these challenges, we propose an input-aware Transformer framework that leverages Semantics and Depth as priors (named SDTR). Our approach involves the use of an S-D Encoder that explicitly models semantic and depth priors, thereby disentangling the learning process of object categorization and position estimation. Moreover, we introduce a Prior-guided Query Builder that incorporates the semantic prior into the initial queries of the Transformer, resulting in more effective input-aware queries. Extensive experiments on the nuScenes and Lyft benchmarks demonstrate the state-of-the-art performance of our method in both 3D object detection and BEV segmentation tasks.

The New Ecology of Chinese Language and Literature in the Digital Age--Cultural Inheritance and Innovation

Abstract Digital media is perpetually evolving, significantly impacting the Chinese language and literature. Concurrently, the rapid pace of modern lifestyles poses challenges to the transmission of cultural heritage within Chinese literary traditions. Accordingly, this study explores the pathways for digital innovation in Chinese literature, focusing on the integration of key virtual technologies to facilitate the cultural inheritance of the Chinese language. This research highlights the application of advanced virtual technologies in preserving and transmitting Chinese linguistic culture. It involves designing an experiential process for cultural inheritance, utilizing these technologies to process graphic and textual information pertinent to the Chinese language and literature. Furthermore, the study innovates cultural transmission methods through the synthesis of virtual 3D scenes and the implementation of human-computer interaction functionalities. An experimental evaluation approach was employed to assess the impact of virtual reality technology on the learning outcomes in Chinese literature and its cultural transmission. The findings indicate that users who interactively engage with the content in a virtual environment achieve a learning effect score that is 0.6528 points higher than those using traditional browsing methods. This underscores the enhanced efficacy of interactive experiences in facilitating the learning of Chinese literature. The regression analysis further demonstrates the significant promotional effects of the number of virtual experiences in Chinese literature (P1) and the rate of user interaction with computers (P2) on the dissemination of Chinese cultural content. The fixed and random effects for the number of virtual experiences are 0.9428 and 0.9784, respectively, while those for user interaction rates are 0.8945 and 0.8835. These results suggest a substantial impact on the broader dissemination and transmission of Chinese language literature, reinforcing the value of integrating virtual reality technologies in cultural education.

Camera-Based 3D Semantic Scene Completion With Sparse Guidance Network.

Semantic scene completion (SSC) aims to predict the semantic occupancy of each voxel in the entire 3D scene from limited observations, which is an emerging and critical task for autonomous driving. Recently, many studies have turned to camera-based SSC solutions due to the richer visual cues and cost-effectiveness of cameras. However, existing methods usually rely on sophisticated and heavy 3D models to process the lifted 3D features directly, which are not discriminative enough for clear segmentation boundaries. In this paper, we adopt the dense-sparse-dense design and propose a one-stage camera-based SSC framework, termed SGN, to propagate semantics from the semantic-aware seed voxels to the whole scene based on spatial geometry cues. Firstly, to exploit depth-aware context and dynamically select sparse seed voxels, we redesign the sparse voxel proposal network to process points generated by depth prediction directly with the coarse-to-fine paradigm. Furthermore, by designing hybrid guidance (sparse semantic and geometry guidance) and effective voxel aggregation for spatial geometry cues, we enhance the feature separation between different categories and expedite the convergence of semantic propagation. Finally, we devise the multi-scale semantic propagation module for flexible receptive fields while reducing the computation resources. Extensive experimental results on the SemanticKITTI and SSCBench-KITTI-360 datasets demonstrate the superiority of our SGN over existing state-of-the-art methods. And even our lightweight version SGN-L achieves notable scores of 14.80% mIoU and 45.45% IoU on SeamnticKITTI validation with only 12.5 M parameters and 7.16 G training memory. Code is available at https://github.com/Jieqianyu/SGN.

You Only Need One Thing One Click: Self-Training for Weakly Supervised 3D Scene Understanding

Understanding 3D scenes, such as semantic segmentation and instance identification within point clouds, typically demands extensive annotated datasets. However, generating point-by-point labels is an overly laborious process. While recent techniques have been developed to train 3D networks with a minimal fraction of labeled points, our method, dubbed “One Thing One Click,” simplifies this by requiring just a single label per object. To effectively utilize these sparse annotations during network training, we’ve crafted an innovative self-training strategy. This involves alternating between training phases and label spreading, powered by a graph propagation module. Additionally, we integrate a relation network to create category-specific prototypes, improving pseudo label accuracy and steering the training process. Our approach also seamlessly integrates with 3D instance segmentation, incorporating a point-clustering technique. Our method demonstrates superior performance over other weakly supervised strategies for 3D semantic and instance segmentation, as evidenced by tests on both ScanNet-v2 and S3DIS datasets. Remarkably, the efficacy of our self-training method with limited annotations rivals that of fully supervised models. Codes and models are available at https://github.com/liuzhengzhe/One-Thing-One-Click .

3D Snapshot: Invertible Embedding of 3D Neural Representations in a Single Image.

3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5MB), and embedding each 3D scene takes up only 160KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse.

SLS4D: Sparse Latent Space for 4D Novel View Synthesis.

Neural radiance fields (NeRF) have achieved great success in novel view synthesis and 3D representation for static scenarios. Existing dynamic NeRFs usually exploit a locally dense grid to fit the deformation fields; however, they fail to capture the global dynamics and concomitantly yield models of heavy parameters. We observe that the 4D space is inherently sparse. Firstly, the deformation fields are sparse in spatial but dense in temporal due to the continuity of motion. Secondly, the radiance fields are only valid on the surface of the underlying scene, usually occupying a small fraction of the whole space. We thus represent the 4D scene using a learnable sparse latent space, a.k.a. SLS4D. Specifically, SLS4D first uses dense learnable time slot features to depict the temporal space, from which the deformation fields are fitted with linear multi-layer perceptions (MLP) to predict the displacement of a 3D position at any time. It then learns the spatial features of a 3D position using another sparse latent space. This is achieved by learning the adaptive weights of each latent feature with the attention mechanism. Extensive experiments demonstrate the effectiveness of our SLS4D: It achieves the best 4D novel view synthesis using only about 6% parameters of the most recent work.

DeepMesh: Differentiable Iso-Surface Extraction.

Geometric Deep Learning has recently made striking progress with the advent of continuous deep implicit fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D euclidean grid, resulting in a learnable parameterization that is unlimited in resolution. Unfortunately, these methods are often unsuitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying implicit field. In this work, we remove this limitation and introduce a differentiable way to produce explicit surface mesh representations from Deep Implicit Fields. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying deep implicit field. We exploit this to define DeepMesh - an end-to-end differentiable mesh representation that can vary its topology. We validate our theoretical insight through several applications: Single view 3D Reconstruction via Differentiable Rendering, Physically-Driven Shape Optimization, Full Scene 3D Reconstruction from Scans and End-to-End Training. In all cases our end-to-end differentiable parameterization gives us an edge over state-of-the-art algorithms.

Exploration of Data Scene Characterization and 3D ROC Evaluation for Hyperspectral Anomaly Detection

Whether or not a hyperspectral anomaly detector is effective is determined by two crucial issues, anomaly detectability and background suppressibility (BS), both of which are very closely related to two factors, the datasets used for a selected hyperspectral anomaly detector and detection measures used for its performance evaluation. This paper explores how anomaly detectability and BS play key roles in hyperspectral anomaly detection (HAD). To address these two issues, we investigate three key elements attributed to HAD. One is a selected hyperspectral anomaly detector, and another is the datasets used for experiments. The third one is the detection measures used to evaluate the effectiveness of a hyperspectral anomaly detector. As for hyperspectral anomaly detectors, twelve commonly used anomaly detectors were evaluated and compared. To address the appropriate use of datasets for HAD, seven popular and widely used datasets were studied for HAD. As for the third issue, the traditional area under a receiver operating characteristic (ROC) curve of detection probability—PD versus false alarm probability, PF, (AUC(D,F))—was extended to 3D ROC analysis where a 3D ROC curve was developed to generate three 2D ROC curves from which eight detection measures could be derived to evaluate HAD in all round aspects, including anomaly detectability, BS and joint anomaly detectability and BS. Qualitative analysis showed that many works reported in the literature which claimed that their developed hyperspectral anomaly detectors performed better than other anomaly detectors are actually not true because they overlooked these two issues. Specifically, a comprehensive study via extensive experiments demonstrated that these 3D ROC curve-derived detection measures can be further used to address the various characterizations of different data scenes and also to provide explanations as to why certain data scenes are not suitable for HAD.

A system for finding identical geometries when building 3-D models

In today's fast-paced world of technology, where innovation plays a crucial role in our daily lives, the importance of search engines for identical models cannot be overstated. Every day, technological progress is becoming an integral part of our existence, penetrating various aspects of our lives. In this context, the search and identification of identical or similar 3D models become particularly relevant and become an integral part of innovative technologies. This article is an in-depth study in the field of creating and describing a mathematical model to search for similar 3D models. Developed using advanced comparison algorithms, this model has enormous potential and can be used in a variety of fields. It is important to emphasize that it is not limited to just one area of application, but finds its application in many areas of human activity. The application of such algorithms is important in various fields, including automated manufacturing, where accuracy and speed of component identification and classification are critical. They also find use in archaeology, where they help in the analysis and study of artifacts and structures. Architecture is also seeing great benefit from these innovations, allowing for more complex and detailed 3D models to be created for design and visualization. The new approach presented in the paper is based on an optimized matching algorithm that enables accurate recognition of geometric structures associated with 3D models without complex data preprocessing or image enhancement. This is a significant step forward in improving the efficiency and accuracy of identifying similar 3D models, even in low-resolution images. However, the application of this model is not limited to these areas. Its integration into real-time systems allows for the identification and classification of both individual 3D objects and complex 3D scenes created from image or video sequences. This is of great importance for the development of virtual and augmented reality technologies, where a high degree of detail and identification is necessary to create captivating and realistic visual effects. Thus, the presented mathematical model and the corresponding software open up enormous prospects for us in various fields. This is an important step forward in the development of technologies related to computer vision and image processing, as well as in the field of virtual and augmented reality, promoting progress and innovation

Sim2Real: Generative AI to Enhance Photorealism through Domain Transfer with GAN and Seven-Chanel-360°-Paired-Images Dataset.

This work aims at providing a solution to data scarcity by allowing end users to generate new images while carefully controlling building shapes and environments. While Generative Adversarial Networks (GANs) are the most common network type for image generation tasks, recent studies have only focused on RGB-to-RGB domain transfer tasks. This study utilises a state-of-the-art GAN network for domain transfer that effectively transforms a multi-channel image from a 3D scene into a photorealistic image. It relies on a custom dataset that pairs 360° images from a simulated domain with corresponding 360° street views. The simulated domain includes depth, segmentation map, and surface normal (stored in seven-channel images), while the target domain is composed of photos from Paris. Samples come in pairs thanks to careful virtual camera positioning. To enhance the simulated images into photorealistic views, the generator is designed to preserve semantic information throughout the layers. The study concludes with photorealistic-generated samples from the city of Paris, along with strategies to further refine model performance. The output samples are realistic enough to be used to train and improve future AI models.

Research Progress of Large Viewing‐Area 3D Holographic Near‐Eye Display

AbstractWith the development of the information industry, the wearable three‐dimensional (3D) near‐eye display that can provide complete visual information has attracted increasing attention. Compared with other existing 3D display technologies, the holographic display is a promising true 3D display technology that can fully record and reproduce the original 3D scenes. The ability to present a continuous parallax and depth perception gives holography technology an unprecedented promise in the next‐generation 3D near‐eye display. However, currently available holographic near‐eye display systems can only provide a limited viewing area (including the field of view and eyebox), greatly restricting its practical application. Focusing on the development requirements of large viewing‐area 3D holographic near‐eye display, the reasons are introduced first why the viewing area of current holographic displays is limited and then the existing solutions from different aspects, including multiplexing‐based methods, steering viewing window methods, aperiodic wavefront modulation methods, and some emerging holographic display technologies are summarized. These various solutions are analyzed respectively, and their representative works are investigated. In addition, the different applications of holography in the near‐eye display system are also discussed. Finally, the advantages and disadvantages of these techniques are compared and perspectives are given.

SAE3D: Set Abstraction Enhancement Network for 3D Object Detection Based Distance Features.

With the increasing demand from unmanned driving and robotics, more attention has been paid to point-cloud-based 3D object accurate detection technology. However, due to the sparseness and irregularity of the point cloud, the most critical problem is how to utilize the relevant features more efficiently. In this paper, we proposed a point-based object detection enhancement network to improve the detection accuracy in the 3D scenes understanding based on the distance features. Firstly, the distance features are extracted from the raw point sets and fused with the raw features regarding reflectivity of the point cloud to maximize the use of information in the point cloud. Secondly, we enhanced the distance features and raw features, which we collectively refer to as self-features of the key points, in set abstraction (SA) layers with the self-attention mechanism, so that the foreground points can be better distinguished from the background points. Finally, we revised the group aggregation module in SA layers to enhance the feature aggregation effect of key points. We conducted experiments on the KITTI dataset and nuScenes dataset and the results show that the enhancement method proposed in this paper has excellent performance.

Algorithm for Creating 3d Scenes of Recognized Objects from Depth Maps

Purpose of research. Development of an algorithm for constructing 3d scenes of recognized objects from synthesized depth maps in order to improve the speed of real-time image processing.Methods. The 3d scene construction algorithm is based on the method of stereo image construction using a threelevel fuzzy depth map construction model. At the first level of this model the boundaries of objects are determined using a modified Canny algorithm, at the second level the values of disparity are calculated on the basis of the sum of absolute differences algorithm modified by fuzzy logic methods, and at the final level the gradients of distances from the boundaries of images to the edges of recognized objects are calculated first and then according to the obtained values of disparity at the second and third levels of the fuzzy hierarchical model, the refined values of disparity are calculated, which are used to carry out the analysis of the depth map.Results. An algorithm for constructing 3d scenes of recognized objects using synthesized depth maps has been developed. It was determined that the proposed algorithm has better performance compared to existing depth map algorithms such as conjugate point algorithm and pyramidal algorithm.Conclusion. The experimental results showed that the proposed algorithm has a lower complexity compared to the analyzed algorithms (conjugate points and pyramidal). The minimum average execution time of the 3d scene construction operation was about 1-2 minutes, which is almost 120 times better compared to the conjugate point algorithms.

SPECIAL EFFECTS USED IN CREATING 3D ANIMATED SCENES PART II

Today, the computers have made special effects so real, very hard to distinguish from the real elements on the screen, that we can barely notice them. The 3D animation technology development mirrors the progress of technology in the 20th century from mechanization and industrial to computerization and virtual. While the special 3D animation from today could not exist without computers, people are still the ones who conceive these effects. There are many opinions according to which 3D animation will have a much greater impact than it has had up to now. Filming animated 3D scenes offers various opportunities for achieving direct camera shooting. 3D Graphics is a lot more complex, it works with textures, volumes, light schemes and everything is being realized with the help of the computer and of the intelligent mind of the 3D graphics editor. Also, the computer-aided design, the integrated machine-tool systems with automatic control, industrial robots, etc., are some of the current mutations of industrial production that strongly influence product design. The 3D programs are spectacular; thanks to the developments from the hard and software technology, lighting and dynamic renderings specific to 3D scenes can be integrated more and more easily even in complex scenes. The viewer will not even notice that some of the nature landscapes will be generated by the 3D graphic editor. The aim of this article is to briefly present the geometric modelling of some component elements of the proposed 3D scene. Like any other developed product, it has to replace one that already exists, and for it to be accomplished an essential criterion is to introduce a new user facility or to improve the technical performance of the predecessor. Competition also led to diversity, the producers had to capture unexplored market sectors and bring improvements specific to a certain job. The aesthetic valences are ensured by the use of high-quality plastic, metal and ceramic materials and finishing, but also by an arrangement of different primary or combined geometric shapes that confer the status of a “decor” object, when we look at the design of new products as an art.

Rethinking scene representation: A saliency-driven hierarchical multi-scale resampling for RGB-D scene point cloud in robotic applications

Effective three-dimensional (3D) scene representation for grasping is significant in smart manufacturing and industrial applications. Serving as a foundational element for robot manipulation, the desired 3D scene representation should encapsulate critical high-level properties. Nonetheless, little attention has been paid to the impact of the object-aware sampling strategy on generating valid grasps. Although pleasing to the eye, natural point cloud mapping is not always appropriate for real-world robotic applications. We introduce a point cloud framework called saliency-driven hierarchical multi-scale resampling: a 3D stereoscopic scene representation that recognizes and tracks novel objects while predicting their saliency in robot interactions. In particular, the new scene representation captures object-level features, which have long been argued as crucial to human scene understanding. Through this approach, we have discovered that our method yields favorable results in terms of feature persistence and resistance to uneven distribution of input scene point clouds, even when the computational time is comparable or slightly increased. Real-world experiments were also conducted to validate the proposed work’s performance to achieve more accurate grasp estimates in generic object and industrial part grasp tasks.

INSTANCE SEGMENTATION OF 3D MESH MODEL BY INTEGRATING 2D AND 3D DATA

Abstract. Buildings are an important part of the urban scene. In this paper, a novel instance segmentation framework for 3D mesh models in urban scenes is proposed. Unlike existing works focusing on semantic segmentation of urban scenes, this work focuses on detecting and segmenting 3D building instances even if they are attached and occluded in a large and imprecise 3D surface model. Multi-view images are first enhanced to RGBH images by adding a height map and are segmented to obtain all roof instances using Mask R-CNN. The 2D roof instances are then back-projected onto the 3D scene, the accurate 3D roof instances are obtained using a novel 3D clustering method and two post-processing steps which preserve the largest connected region and remove the model ambiguity. Finally, the 2D convex hull of each 3D roof instance is calculated and the model is divided within the range into building instances. The performance of the proposed methods is evaluated using real UAV images and the corresponding 3D mesh models qualitatively and quantitatively. Results revealed that the proposed method could effectively segment the model of the urban scenes and building instance is obtained, the over-segmentation masks can be clustered correctly into roof instances and the under-segmentation masks caused by image segmentation errors are eliminated.

3D Monte Carlo surface-atmosphere radiative transfer modelling with DART

Significant errors can arise if the adjacency effect (i.e., the contribution of neighbouring pixels to the radiance of a pixel) is neglected in the interpretation of remote sensing images. For example, adjacency radiances can account for >30% of the signal at the top of the atmosphere (TOA) of a white sand-lined coastline (Bulgarelli and Zibordi, 2018). This paper shows that 3D radiative transfer (RT) modelling can quantify this phenomenon. It presents a new 3D Monte Carlo surface-atmosphere RT modelling in the DART RT model, and a resulting virtual 3D Earth-Atmosphere laboratory for accurate simulation of atmospheric RT, including the adjacency effect. It was first validated with the atmosphere model SMART-G for 2D scenes: relative difference is 0.20% in TOA directional reflectance of infinite black surface for solar zenith equal to 60°, and 0.03% in TOA nadir reflectance of a black disc inside infinite white Lambertian surface, for solar zenith equal to 0° and 30°. Then, the adjacency effect on the TOA radiance and TOA albedo of a 3D scene was studied for a circular city (radius 2 km) surrounded by a forest (dimension 10 km), for four Sentinel-2A bands (blue, green, red, near infrared). Its contribution to TOA nadir radiance reaches ∼20% in the near infrared band, and increases with viewing zenith angle (VZA): ∼60% if VZA = 80°. The 3D scene TOA albedo was calculated after adapting the DART Bidirectional Reflectance Factor (BRF) camera to simulate TOA radiance for all upward directions at any angular resolution. For Sentinel-2A’s four bands, the adjacency effect influences the TOA albedo of the circular city by up to 10% for the green band and up to 27% for the near infrared band, well above the maximum uncertainty 5% usually required in land surface applications. The adjacency effect of the city neighbourhood 3D structure was studied by replacing it by a Lambertian surface with its albedo. The city nadir TOA radiance changed by up to 1.3% with very small change in TOA albedo (<0.5%). This new modelling greatly improves DART potential for accurate simulation of atmospheric effects. It is in the DART version freely available for research and education (https://dart.omp.eu).

Preface: Workshop “Semantics3D - Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences”

Preface: Workshop “Semantics3D - Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences”

Preface: Workshop “Semantics3D - Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences”

Preface: Workshop “Semantics3D - Semantic Scene Analysis and 3D Reconstruction from Images and Image Sequences”

AN EVALUATION OF STEREO AND MULTIVIEW ALGORITHMS FOR 3D RECONSTRUCTION WITH SYNTHETIC DATA

Abstract. The reconstruction of 3D scenes from images has usually been addressed with two different strategies, namely stereo and multiview. The former requires rectified images and generates a disparity map, while the latter relies on the camera parameters and directly retrieves a depth map. For both cases, deep learning architectures have shown an outstanding performance. However, due to the differences between input and output data, the two strategies are difficult to compare on a common scene. Moreover, for remote sensing applications multi-view data is hard to acquire and the ground truth is either sparse or affected by outliers. Hence, in this article we evaluate the performance of stereo and multi-view architectures trained on synthetic data resembling remote sensing images. The data has been and processed and organized to be compatible with both kind of neural networks. For a fair comparison, training and testing are done only with two views. We focus on the accuracy of the reconstruction, as well as the impact of the depth range and the baseline of the stereo array. Results are presented for deep learning architectures and non-learning algorithms.