Large Scenes Research Articles

Fast Building Instance Proxy Reconstruction for Large Urban Scenes.

Digitalization of large-scale urban scenes (in particular buildings) has been a long-standing open problem, which attributes to the challenges in data acquisition, such as incomplete scene coverage, lack of semantics, low efficiency, and low reliability in path planning. In this paper, we address these challenges in urban building reconstruction from aerial images, and we propose an effective workflow and a few novel algorithms for efficient 3D building instance proxy reconstruction for large urban scenes. Specifically, we propose a novel learning-based approach to instance segmentation of urban buildings from aerial images followed by a voting-based algorithm to fuse the multi-view instance information to a sparse point cloud (reconstructed using a standard Structure from Motion pipeline). Our method enables effective instance segmentation of the building instances from the point cloud. We also introduce a layer-based surface reconstruction method dedicated to the 3D reconstruction of building proxies from extremely sparse point clouds. Extensive experiments on both synthetic and real-world aerial images of large urban scenes have demonstrated the effectiveness of our approach. The generated scene proxy models can already provide a promising 3D surface representation of the buildings in large urban scenes, and when applied to aerial path planning, the instance-enhanced building proxy models can significantly improve data completeness and accuracy, yielding highly detailed 3D building models.

Efficient Rendering of Digital Twins Consisting of Both Static And Dynamic Data

Abstract. The high number of vertices and faces present in a digital twin of a large scene poses a challenge when executing large-scale simulations that require the data to be rendered multiple times with slight changes such as the light conditions in a day-night cycle during a long period of time (weeks and even months). Adding urban morphology such as tree planting introduces a new layer of complexity due to the requirement for evaluating different tree configurations and their effect on the local environment in the context of urban heat islands. Using advanced rendering techniques such as splitting a single buffer into sub-regions with inplace updates as well as buffer orphaning multiple methodologies are evaluated.

Robust and accurate visual geo-localization using prior map constructed by handheld LiDAR SLAM with camera image and terrestrial LiDAR point cloud

ABSTRACT Visual geo-localization using prior map of known environments has extensive applications in the fields such as self-driving, augmented reality and navigation. Currently, such prior maps are usually constructed by visual SLAM or SFM. However, the recent advances in LiDAR SLAM technology with RGB camera demonstrate great potential in efficient and accurate prior map construction and visual geo-localization. In this research, we developed a novel prior map construction approach which seamlessly integrates a handheld LiDAR SLAM system, a global LiDAR localization algorithm and a sparse set of terrestrial LiDAR scans as accurate control within a factor graph optimization framework. The developed LiDAR SLAM system augmented by terrestrial LiDAR scans achieves an improved mapping accuracy of 8 cm under GNSS-denied conditions in the areas over 50,000 m2 while the state-of-art FAST-LIO2 alone fails and produces a mapping error over 30 m. The results show that a competitive accuracy of visual geo-localization on a mobile phone using the constructed prior map is achieved (i.e. 60 and 76 cm in two tests respectively). Finally, this work also demonstrates a novel LiDAR point cloud fusion method that produces optimized and consistent coarse-to-fine 3D reconstruction in large and complex scenes.

FaSS-MVS: Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-Borne Monocular Imagery.

With FaSS-MVS, we present a fast, surface-aware semi-global optimization approach for multi-view stereo that allows for rapid depth and normal map estimation from monocular aerial video data captured by unmanned aerial vehicles (UAVs). The data estimated by FaSS-MVS, in turn, facilitate online 3D mapping, meaning that a 3D map of the scene is immediately and incrementally generated as the image data are acquired or being received. FaSS-MVS is composed of a hierarchical processing scheme in which depth and normal data, as well as corresponding confidence scores, are estimated in a coarse-to-fine manner, allowing efficient processing of large scene depths, such as those inherent in oblique images acquired by UAVs flying at low altitudes. The actual depth estimation uses a plane-sweep algorithm for dense multi-image matching to produce depth hypotheses from which the actual depth map is extracted by means of a surface-aware semi-global optimization, reducing the fronto-parallel bias of Semi-Global Matching (SGM). Given the estimated depth map, the pixel-wise surface normal information is then computed by reprojecting the depth map into a point cloud and computing the normal vectors within a confined local neighborhood. In a thorough quantitative and ablative study, we show that the accuracy of the 3D information computed by FaSS-MVS is close to that of state-of-the-art offline multi-view stereo approaches, with the error not even an order of magnitude higher than that of COLMAP. At the same time, however, the average runtime of FaSS-MVS for estimating a single depth and normal map is less than 14% of that of COLMAP, allowing us to perform online and incremental processing of full HD images at 1-2 Hz.

Mesoscale explorer: Visual exploration of large-scale molecular models.

The advent of cryo-electron microscopy (cryo-EM) and cryo-electron tomography (cryo-ET), coupled with computational modeling, has enabled the creation of integrative 3D models of viruses, bacteria, and cellular organelles. These models, composed of thousands of macromolecules and billions of atoms, have historically posed significant challenges for manipulation and visualization without specialized molecular graphics tools and hardware. With the recent advancements in GPU rendering power and web browser capabilities, it is now feasible to render interactively large molecular scenes directly on the web. In this work, we introduce Mesoscale Explorer, a web application built using the Mol* framework, dedicated to the visualization of large-scale molecular models ranging from viruses to cell organelles. Mesoscale Explorer provides unprecedented access and insight into the molecular fabric of life, enhancing perception, streamlining exploration, and simplifying visualization of diverse data types, showcasing the intricate details of these models with unparalleled clarity.

HiSC4D: Human-Centered Interaction and 4D Scene Capture in Large-Scale Space Using Wearable IMUs and LiDAR.

We introduce HiSC4D, a novel Human-centered interaction and 4D Scene Capture method, aimed at accurately and efficiently creating a dynamic digital world, containing large-scale indoor-outdoor scenes, diverse human motions, rich human-human interactions, and human-environment interactions. By utilizing body-mounted IMUs and a head-mounted LiDAR, HiSC4D can capture egocentric human motions in unconstrained space without the need for external devices and pre-built maps. This affords great flexibility and accessibility for human-centered interaction and 4D scene capturing in various environments. Taking into account that IMUs can capture human spatially unrestricted poses but are prone to drifting for long-period using, and while LiDAR is stable for global localization but rough for local positions and orientations, HiSC4D employs a joint optimization method, harmonizing all sensors and utilizing environment cues, yielding promising results for long-term capture in large scenes. To promote research of egocentric human interaction in large scenes and facilitate downstream tasks, we also present a dataset, containing 8 sequences in 4 large scenes (200 to 5,000 m2 ), providing 36k frames of accurate 4D human motions with SMPL annotations and dynamic scenes, 31k frames of cropped human point clouds, and scene mesh of the environment. A variety of scenarios, such as the basketball gym and commercial street, alongside challenging human motions, such as daily greeting, one-on-one basketball playing, and tour guiding, demonstrate the effectiveness and the generalization ability of HiSC4D. The dataset and code will be publicly available for research purposes.

About the Possibility of Using Fixation Microsaccades to Improve the Quality of Visible Images in the Foveal Zone

The article is devoted to the description and analysis of a computer model that was created by D. S. Lebedev to demonstrate the possibility of a positive effect of fixation microsaccadic eye movements on the perception of small stimuli. The model is based on the assumption that in the process of fixing the gaze on the test stimulus, several “neural images” of this stimulus, resulting from microsaccades, are summed up in the brain. The series of summed neural images correspond to a sequence of shifted positions of the optical image of a stimulus on the retina. To accurately superimpose neural images on each other, a mechanism for compensating fixation saccadic microshifts is introduced into the model, identical to the mechanism that ensures the constancy of spatial perception in the case of macrosaccades, i.e. when turning the eyes to view large objects or scenes. The author of the model assessed the possibility of improving the quality of visible images by increasing the signal-to-noise ratio, which can be achieved using realistic spatiotemporal parameters of test images, neural noise and eye micromovements, selected bу means of literature analysis. Results of model calculation obtained for the specific parameters of the retina and eye movements showed that the considered summation mechanism with compensation for saccadic shifts can progressively improve the quality of visible test stimuli when the number of summed neural images increases to approximately seven or eight, after which the positive effect practically does not increase. In this article, based on the material of recordings of eye movements in relevant experiments, the degree of realism of this model is discussed.

Multi-Scale Classification and Contrastive Regularization: Weakly Supervised Large-Scale 3D Point Cloud Semantic Segmentation

With the proliferation of large-scale 3D point cloud datasets, the high cost of per-point annotation has spurred the development of weakly supervised semantic segmentation methods. Current popular research mainly focuses on single-scale classification, which fails to address the significant feature scale differences between background and objects in large scenes. Therefore, we propose MCCR (Multi-scale Classification and Contrastive Regularization), an end-to-end semantic segmentation framework for large-scale 3D scenes under weak supervision. MCCR first aggregates features and applies random downsampling to the input data. Then, it captures the local features of a random point based on multi-layer features and the input coordinates. These features are then fed into the network to obtain the initial and final prediction results, and MCCR iteratively trains the model using strategies such as contrastive learning. Notably, MCCR combines multi-scale classification with contrastive regularization to fully exploit multi-scale features and weakly labeled information. We investigate both point-level and local contrastive regularization to leverage point cloud augmentor and local semantic information and introduce a Decoupling Layer to guide the loss optimization in different spaces. Results on three popular large-scale datasets, S3DIS, SemanticKITTI and SensatUrban, demonstrate that our model achieves state-of-the-art (SOTA) performance on large-scale outdoor datasets with only 0.1% labeled points for supervision, while maintaining strong performance on indoor datasets.

Multi-Robot Collaborative Mapping with Integrated Point-Line Features for Visual SLAM.

Simultaneous Localization and Mapping (SLAM) enables mobile robots to autonomously perform localization and mapping tasks in unknown environments. Despite significant progress achieved by visual SLAM systems in ideal conditions, relying solely on a single robot and point features for mapping in large-scale indoor environments with weak-texture structures can affect mapping efficiency and accuracy. Therefore, this paper proposes a multi-robot collaborative mapping method based on point-line fusion to address this issue. This method is designed for indoor environments with weak-texture structures for localization and mapping. The feature-extraction algorithm, which combines point and line features, supplements the existing environment point feature-extraction method by introducing a line feature-extraction step. This integration ensures the accuracy of visual odometry estimation in scenes with pronounced weak-texture structure features. For relatively large indoor scenes, a scene-recognition-based map-fusion method is proposed in this paper to enhance mapping efficiency. This method relies on visual bag of words to determine overlapping areas in the scene, while also proposing a keyframe-extraction method based on photogrammetry to improve the algorithm's robustness. By combining the Perspective-3-Point (P3P) algorithm and Bundle Adjustment (BA) algorithm, the relative pose-transformation relationships of multi-robots in overlapping scenes are resolved, and map fusion is performed based on these relative pose relationships. We evaluated our algorithm on public datasets and a mobile robot platform. The experimental results demonstrate that the proposed algorithm exhibits higher robustness and mapping accuracy. It shows significant effectiveness in handling mapping in scenarios with weak texture and structure, as well as in small-scale map fusion.

Mesoscale Explorer - Visual Exploration of Large-Scale Molecular Models.

The advent of cryo-electron microscopy (cryo-EM) and cryo-electron tomography (cryo-ET), coupled with computational modeling, has enabled the creation of integrative 3D models of viruses, bacteria, and cellular organelles. These models, composed of thousands of macromolecules and billions of atoms, have historically posed significant challenges for manipulation and visualization without specialized molecular graphics tools and hardware. With the recent advancements in GPU rendering power and web browser capabilities, it is now feasible to render interactively large molecular scenes directly on the web. In this work, we introduce Mesoscale Explorer, a web application built using the Mol* framework, dedicated to the visualization of large-scale molecular models ranging from viruses to cell organelles. Mesoscale Explorer provides unprecedented access and insight into the molecular fabric of life, enhancing perception, streamlining exploration, and simplifying visualization of diverse data types, showcasing the intricate details of these models with unparalleled clarity.

Dense Feature Pyramid Deep Completion Network

Most current point cloud super-resolution reconstruction requires huge calculations and has low accuracy when facing large outdoor scenes; a Dense Feature Pyramid Network (DenseFPNet) is proposed for the feature-level fusion of images with low-resolution point clouds to generate higher-resolution point clouds, which can be utilized to solve the problem of the super-resolution reconstruction of 3D point clouds by turning it into a 2D depth map complementation problem, which can reduce the time and complexity of obtaining high-resolution point clouds only by LiDAR. The network first utilizes an image-guided feature extraction network based on RGBD-DenseNet as an encoder to extract multi-scale features, followed by an upsampling block as a decoder to gradually recover the size and details of the feature map. Additionally, the network connects the corresponding layers of the encoder and decoder through pyramid connections. Finally, experiments are conducted on the KITTI deep complementation dataset, and the network performs well in various metrics compared to other networks. It improves the RMSE by 17.71%, 16.60%, 7.11%, and 4.68% compared to the CSPD, Spade-RGBsD, Sparse-to-Dense, and GAENET.

OccFaçade: enabling precise building façade parsing in large urban scenes with occlusion

ABSTRACT Building façade parsing is to recognize the building façade image into different categories of individuals including walls, doors, windows, balconies, etc. However, obstructions such as trees present a significant challenge to conducting façade parsing. In this paper, we designed OccFaçade to achieve high-precision parsing of occluded building façades in large urban scenes. OccFaçade primarily incorporates two modules, Multi-layer Dilated Convolution Module (MD-Module) and Multi-scale Row-Column Convolution Module (MRC-Module), to capture repeated texture in local and row-column directions. This aims to leverage repetitive textures to address occlusion challenges in building façade parsing. Besides, we introduce our building façade dataset MeshFaçade from the Mesh data generated by drone imagery to study the occlusion problem of missing textures. The experimental results demonstrate that OccFaçade achieves state-of-the-art performance with mIOU of 85.01%, 84.09%, 72.95%, and 88.83% on the ENPC2014 dataset, ECP dataset, RueMonge2014 dataset, and our MeshFaçade dataset, respectively. The code and data are available at https://github.com/yueyisui/OccFacade.

Design of railway simulation training system based on WebGL

Abstract In order to solve the problems of poor interaction with students and insufficient practicality in the previous railway freight station operation training, this article developed a training system based on WebGL technology that can interact with students and truly display the operation situation of the freight yard. This system is based on the Unity3D engine and WebGL technology. It imports 3D models to build large map scenes, and introduces functions such as cargo bundling and map roaming. The complete designed system is published to the Web through WebGL technology. After experimental analysis, the parking lot training system has solved the problems of poor practicality, little interaction, low efficiency, and narrow coverage in previous teaching. It has promoted the standardization and standardization of training students’ future work, and has effectively improved the professionalism of parking lot managers.

3D dynamic landscape simulation of artificial intelligence in environmental landscape design

Three-dimensional (3D) simulations and precise landscape visualizations are crucial for various applications, like landscape management and planning, computer and connection of the landscape, evaluation, and tracking of land use. The consequences of several plans and a large scene cannot be communicated using older methods of comprehensive environmental planning and development in a timely, rational, and coordinated manner. Architects have trouble incorporating ideas into other comprehensive planning implementation processes. Architects did not thoroughly investigate the neighbourhood's demographics and matching behavioural needs and lacked critical thinking. The 3D dynamic landscape simulation is a detailed computerized three-dimensional simulation of the environment that can be dynamically presented. With the aid of Artificial Intelligence (AI) technology, the system possesses a strong sense of reality, a user-friendly interface, and interactive features that can be tailored to the requirements of the contemporary urban environmental landscape. Regarding exterior publicity, domestic assistance, environmental land use planning, and information systems. The novelty of the proposed Interactive Design System based on AI (IDS-AI) is to create a 3D dynamic landscape model based on a real-life environmental scene, utilizing a Geographic Information System (GIS) to optimize landscape vision. Secondly, 3D environmental landscape design simulation was implemented using GIS spatial analysis in conjunction with the Fuzzy Analytical Hierarchical Process (FAHP) to reduce the data overlap rate and help make an accurate decision. Finally, the design incorporates the development of the interactive interface system application of landscape design and environmental resources for viewing the landscape, the factors that affect them, and the area coverage ratio of various land cover types. The experimental outcomes show that the suggested IDS model increases the gradient sensitivity level of 98.3 % and area coverage ratio of 93.4 % compared to other existing models.

A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets

Novel view synthesis has seen major advances in recent years, with 3D Gaussian splatting offering an excellent level of visual quality, fast training and real-time rendering. However, the resources needed for training and rendering inevitably limit the size of the captured scenes that can be represented with good visual quality. We introduce a hierarchy of 3D Gaussians that preserves visual quality for very large scenes, while offering an efficient Level-of-Detail (LOD) solution for efficient rendering of distant content with effective level selection and smooth transitions between levels. We introduce a divide-and-conquer approach that allows us to train very large scenes in independent chunks. We consolidate the chunks into a hierarchy that can be optimized to further improve visual quality of Gaussians merged into intermediate nodes. Very large captures typically have sparse coverage of the scene, presenting many challenges to the original 3D Gaussian splatting training method; we adapt and regularize training to account for these issues. We present a complete solution, that enables real-time rendering of very large scenes and can adapt to available resources thanks to our LOD method. We show results for captured scenes with up to tens of thousands of images with a simple and affordable rig, covering trajectories of up to several kilometers and lasting up to one hour.

SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration

Recent techniques for real-time view synthesis have rapidly advanced in fidelity and speed, and modern methods are capable of rendering near-photorealistic scenes at interactive frame rates. At the same time, a tension has arisen between explicit scene representations amenable to rasterization and neural fields built on ray marching, with state-of-the-art instances of the latter surpassing the former in quality while being prohibitively expensive for real-time applications. We introduce SMERF, a view synthesis approach that achieves state-of-the-art accuracy among real-time methods on large scenes with footprints up to 300 m 2 at a volumetric resolution of 3.5 mm 3 . Our method is built upon two primary contributions: a hierarchical model partitioning scheme, which increases model capacity while constraining compute and memory consumption, and a distillation training strategy that simultaneously yields high fidelity and internal consistency. Our method enables full six degrees of freedom navigation in a web browser and renders in real-time on commodity smartphones and laptops. Extensive experiments show that our method exceeds the state-of-the-art in real-time novel view synthesis by 0.78 dB on standard benchmarks and 1.78 dB on large scenes, renders frames three orders of magnitude faster than state-of-the-art radiance field models, and achieves real-time performance across a wide variety of commodity devices, including smartphones. We encourage readers to explore these models interactively at our project website: https://smerf-3d.github.io.

BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

We present BlockFusion, a diffusion-based model that generates 3D scenes as unit blocks and seamlessly incorporates new blocks to extend the scene. BlockFusion is trained using datasets of 3D blocks that are randomly cropped from complete 3D scene meshes. Through per-block fitting, all training blocks are converted into the hybrid neural fields: with a tri-plane containing the geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the signed distance values. A variational auto-encoder is employed to compress the tri-planes into the latent tri-plane space, on which the denoising diffusion process is performed. Diffusion applied to the latent representations allows for high-quality and diverse 3D scene generation. To expand a scene during generation, one needs only to append empty blocks to overlap with the current scene and extrapolate existing latent tri-planes to populate new blocks. The extrapolation is done by conditioning the generation process with the feature samples from the overlapping tri-planes during the denoising iterations. Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that harmoniously blend with the existing scene. A 2D layout conditioning mechanism is used to control the placement and arrangement of scene elements. Experimental results indicate that BlockFusion is capable of generating diverse, geometrically consistent and unbounded large 3D scenes with unprecedented high-quality shapes in both indoor and outdoor scenarios.

Embedded System Based on Task Interval Data Interaction Model

In the Internet of Things era, more intelligent systems can communicate with each other. Embedded system combined with network communication applications has become the basis for Internet of Things research. The programmable logic unit designed by ARM architecture has great advantages in running speed, power control and so on. In the paper, to solve the problems of common embedded MCU control resources occupying a large amount of memory and the slow speed of building engineering simulation model, it is necessary to use the queuing connection algorithm model to directly input the timing physical characteristics of the code stream in the embedded system to calculate and match the timing physical characteristics of the output code stream. The optimization algorithm of DTF-MARTE to detect the probability of timing deviation is used in the paper. It is to detect the problem of inaccurate timing information in the demand. We compare the expected physical characteristics of the timing sequence and obtain the timing deviation probability of the output data stream. The model developed in this paper has the characteristic of dynamic reconfiguration of the task interval. The design of monotonically decreasing data tasks can be realized, and the reconfigured task modules are used for interacting the data buffer area and dynamically reconstructing the instruction overhead and transmission. We analyze the performance comparison between the proposed model and the traditional communication connection model. It proves that the proposed model can further improve the priority queue and guide the data flow. According to that method, the problem of asynchronous spatial data interaction by controlling and combining different communication modes in a large scene can be solved. Data interaction can be triggered at a fixed time, and mutual interference of randomly triggered wireless communication and data acquisition modules can be avoided. It can solve the problem of insufficient computing power when future embedded devices need massive data encryption in the Internet of Things era, and provide a new way of thinking for fast, safe and efficient implementation.

Neural Colour Correction for Indoor 3D Reconstruction Using RGB-D Data.

With the rise in popularity of different human-centred applications using 3D reconstruction data, the problem of generating photo-realistic models has become an important task. In a multiview acquisition system, particularly for large indoor scenes, the acquisition conditions will differ along the environment, causing colour differences between captures and unappealing visual artefacts in the produced models. We propose a novel neural-based approach to colour correction for indoor 3D reconstruction. It is a lightweight and efficient approach that can be used to harmonize colour from sparse captures over complex indoor scenes. Our approach uses a fully connected deep neural network to learn an implicit representation of the colour in 3D space, while capturing camera-dependent effects. We then leverage this continuous function as reference data to estimate the required transformations to regenerate pixels in each capture. Experiments to evaluate the proposed method on several scenes of the MP3D dataset show that it outperforms other relevant state-of-the-art approaches.

Intelligently Counting Agricultural Pests by Integrating SAM with FamNet

The utilization of the large pretrained model (LPM) based on Transformer has emerged as a prominent research area in various fields, owing to its robust computational capabilities. However, there remains a need to explore how LPM can be effectively employed in the agricultural domain. This research aims to enhance agricultural pest detection with limited samples by leveraging the strong generalization performance of the LPM. Through extensive research, this study has revealed that in tasks involving the counting of a small number of samples, complex agricultural scenes with varying lighting and environmental conditions can significantly impede the accuracy of pest counting. Consequently, accurately counting pests in diverse lighting and environmental conditions with limited samples remains a challenging task. To address this issue, the present research suggests a unique approach that integrates the outstanding performance of the segment anything model in class-agnostic segmentation with the counting network. Moreover, by intelligently utilizing a straightforward TopK matching algorithm to propagate accurate labels, and drawing inspiration from the GPT model while incorporating the forgetting mechanism, a more robust model can be achieved. This approach transforms the problem of matching instances in different scenarios into a problem of matching similar instances within a single image. Experimental results demonstrate that our method enhances the accuracy of the FamNet baseline model by 69.17% on this dataset. Exploring the synergy between large models and agricultural scenes warrants further discussion and consideration.