Complex Urban Scenes Research Articles

LightUrban: Similarity Based Fine‐grained Instancing for Lightweighting Complex Urban Point Clouds

AbstractLarge‐scale urban point clouds play a vital role in various applications, while rendering and transmitting such data remains challenging due to its large volume, complicated structures, and significant redundancy. In this paper, we present LightUrban, the first point cloud instancing framework for efficient rendering and transmission of fine‐grained complex urban scenes. We first introduce a segmentation method to organize the point clouds into individual buildings and vegetation instances from coarse to fine. Next, we propose an unsupervised similarity detection approach to accurately group instances with similar shapes. Furthermore, a fast pose and size estimation component is applied to calculate the transformations between the representative instance and the corresponding similar instances in each group. By replacing individual instances with their group's representative instances, the data volume and redundancy can be dramatically reduced. Experimental results on large‐scale urban scenes demonstrate the effectiveness of our algorithm. To sum up, our method not only structures the urban point clouds but also significantly reduces data volume and redundancy, filling the gap in lightweighting urban landscapes through instancing.

Combining geometrical and intensity information to recognize vehicles from super-high density UAV-LiDAR point clouds

The traditional airborne LiDAR point clouds vehicle recognition algorithms only utilize the contained geometric information and are not sufficient to recognize vehicles accurately, especially in complex urban scenes. And their applicability in super-high density UAV-LiDAR point clouds is unknown. Therefore, a 3-D algorithm combining geometric and intensity information from super-high density UAV-LiDAR point clouds is presented. The proposed algorithm firstly converts the original point clouds into a 3D multivalued image which fuses intensity, elevation and density information simultaneously. Thereafter, the potential vehicle voxels are extracted according to the intensity, elevation and density consistency of vehicles. Subsequently, individual vehicles are recognized using the potential vehicle voxel’s spatially connected set with vehicle size constraint. Finally, the quantified attribute information of each individual vehicle, containing the spatial location, type, and size, is determined. The results for recognized vehicles are evaluated using UAV-LiDAR data with different densities and demonstrate an average quality (Kappa coefficient) of 96.58% (96.04%) without being significantly affected by occlusion and the very close vehicle arrangement.

Building usage prediction in complex urban scenes by fusing text and facade features from street view images using deep learning

Building usage maps are inputs to many urban planning applications, however, the existing methods and the available data have limitations in generating instance-level high-resolution usage maps. In this study we tackle this problem by utilizing Street View Images (SVIs) and proposing a novel ensemble learning architecture that leverages building facade features and text extracted from hoardings, posters, etc. on buildings to predict the usage class. A pre-trained object detection model i.e., Grounding DINO, is implemented to efficiently identify buildings. A novel manually labeled training data of detected buildings corresponding to their usage is used to extract features from building facades across diverse Indian cities (Hyderabad, Mumbai, Bangalore, Delhi) using Vision Transformer (ViT) model. Following this, CLIPSeg a pre-trained segmentation model is used to recognizes text specifically on building elements like signs, posters, and banners. We then leverage GPT-3.5 Turbo, a Large Language Model (LLM), fine-tuned with a specifically designed few-shot prompting method, to infer building usage from the recognized text. To achieve optimal performance, the proposed ensemble linear metaclassifier combines predictions from ViT and LLM model. The predicted building usages are attributed to their corresponding locations to develop spatial maps. An analysis of our framework compared against ground truth data collected from various Indian cities reveals significantly accurate outcomes. Our findings highlight the utility of textual information in classifying utilities and commercial buildings, while features extracted from vision models prove more informative for residential buildings. Our approach can automate the generation of roadside building attributes and usage details on a larger scale.

Active Reinforcement Learning for the Semantic Segmentation of Urban Images

Image segmentation using supervised learning algorithms usually requires large amounts of annotated training data, while urban datasets frequently contain unbalanced classes leading to poor detection of under-represented classes. We investigate the use of a reinforced active learning method to address the limitations of semantic segmentation on complex urban scenes. In this method, an agent learns to select small informative regions of the image to be labeled from a pool of unlabeled data. The agent is represented by a deep Q-Network, where a Markov Decision Process (MDP) is used to formulate the Active Learning problem. We introduced the Frequency Weighted Average IoU (FWA IoU) as the image region selection performance metric to reduce the amount of training data while achieving competitive results. Using the Cityscapes and GTAv urban datasets, three baseline image segmentation networks (FPN, DeepLabV3, DeepLabV3+) trained with image regions selected by the proposed FWA IoU metric performed better compared to baseline region selection by active learning methods such as the Random selection, Entropy-based selection, and Bayesian Active Learning by Disagreement. Training performance equivalent to 98% of the fully supervised segmentation network was achieved by labeling only 8% of the dataset.

Exploration of an Open Vocabulary Model on Semantic Segmentation for Street Scene Imagery

This study investigates the efficacy of an open vocabulary, multi-modal, foundation model for the semantic segmentation of images from complex urban street scenes. Unlike traditional models reliant on predefined category sets, Grounded SAM uses arbitrary textual inputs for category definition, offering enhanced flexibility and adaptability. The model’s performance was evaluated across single and multiple category tasks using the benchmark datasets Cityscapes, BDD100K, GTA5, and KITTI. The study focused on the impact of textual input refinement and the challenges of classifying visually similar categories. Results indicate strong performance in single-category segmentation but highlighted difficulties in multi-category scenarios, particularly with categories bearing close textual or visual resemblances. Adjustments in textual prompts significantly improved detection accuracy, though challenges persisted in distinguishing between visually similar objects such as buses and trains. Comparative analysis with state-of-the-art models revealed Grounded SAM’s competitive performance, particularly notable given its direct inference capability without extensive dataset-specific training. This feature is advantageous for resource-limited applications. The study concludes that while open vocabulary models such as Grounded SAM mark a significant advancement in semantic segmentation, further improvements in integrating image and text processing are essential for better performance in complex scenarios.

CONNECTING EMBEDDED HISTORY OF THE URBAN INFRASTRUCTURE TO THE CONTEMPORARY CITY: A CASE STUDY OF GWANGHWAMUN WOLDAE RESTORATION

Abstract. The captured 3D digital record of the excavation site represents a momentary state of the city, but also conveys rich evidence and stories as an urban environment created by the interaction of actors, artifacts, and institutions surrounding the place. These digital records combine multiple temporal layers of complex urban scenes and require interpretation by multiple stakeholders for comprehensive decision-making. This study revealed that in the context of Seoul, the Cultural Heritage Administration and the City of Seoul have made significant use of digital records in cultural heritage restoration from the perspective of preservation and utilization. The two organizations collaborated on the Gwanghwamun Woldae excavation survey recently. The site has been dealing with the multi-layered time and space of complex urban sites for over 40 years while carrying out urban planning and cultural heritage preservation. Through a case study of the Gwanghwamun Woldae excavation site, we articulate the nature of governance in digital documentation to extend its application. The goal is to highlight the urban cross-section and historical layers of urban infrastructure captured digitally - LiDAR and photogrammetric scanning - at the Gwanghwamun Woldae excavation site. This involves describing the chronological layers of tram lines, streets and buildings from different eras and linking them to the current urban context of Seoul through analog historical records such as maps and plan diagrams. Ultimately, we discuss the value of digital restoration of technologies, events and landscapes as a record of the process valuing modern urban heritage.

An Improved Method for Individual Tree Segmentation in Complex Urban Scenes Based on Using Multispectral LiDAR by Deep Learning

An Improved Method for Individual Tree Segmentation in Complex Urban Scenes Based on Using Multispectral LiDAR by Deep Learning

Robust Fusion of Multi-Source Images for Accurate 3D Reconstruction of Complex Urban Scenes

Integrated reconstruction is crucial for 3D modeling urban scenes using multi-source images. However, large viewpoint and illumination variations pose challenges to existing solutions. A novel approach for accurate 3D reconstruction of complex urban scenes based on robust fusion of multi-source images is proposed. Firstly, georeferenced sparse models are reconstructed from the terrestrial and aerial images using GNSS-aided incremental SfM, respectively. Then, cross-platform match pairs are selected based on point-on-image observability. The terrestrial and aerial images are robustly matched based on the selected match pairs to generate cross-platform tie points. Thirdly, the tie points are triangulated to derive cross-platform 3D correspondences. The 3D correspondences are refined using a novel outlier detection method. Finally, the terrestrial and aerial sparse models are merged based on the refined correspondences, and the integrated model is globally optimized to obtain an accurate reconstruction of the scene. The proposed methodology is evaluated on five benchmark datasets, and extensive experiments are performed. The proposed pipeline is compared with a state-of-the-art methodology and three widely used software packages. Experimental results demonstrate that the proposed methodology outperforms the other pipelines in terms of robustness and accuracy.

Detection and Classification of Buildings by Height from Single Urban High-Resolution Remote Sensing Images

Recent improvements in remote sensing technologies have boosted building detection techniques from rough classifications using moderate resolution imagery to precise extraction from high-resolution imagery. Shadows frequently emerge in high-resolution urban images. To exploit shadow information, we developed a novel building detection and classification algorithm for images of urban areas with large-size shadows, employing only the visible spectral bands to determine the height levels of buildings. The proposed method, building general-classified by height (BGCH), calculates shadow orientation, detects buildings using seed-blocks, and classifies the buildings into different height groups. Our proposed approach was tested on complex urban scenes from Toronto and Beijing. The experimental results illustrate that our proposed method accurately and efficiently detects and classifies buildings by their height levels; the building detection rate exceeded 95%. The precision of classification by height levels was over 90%. This novel building-height-level detection method provides rich information at low cost and is suitable for further city scene analysis, flood disaster risk assessment, population estimation, and building change detection applications.

Road-Side Individual Tree Segmentation from Urban MLS Point Clouds Using Metric Learning

As one of the most important components of urban space, an outdated inventory of road-side trees may misguide managers in the assessment and upgrade of urban environments, potentially affecting urban road quality. Therefore, automatic and accurate instance segmentation of road-side trees from urban point clouds is an important task in urban ecology research. However, previous works show under- or over-segmentation effects for road-side trees due to overlapping, irregular shapes and incompleteness. In this paper, a deep learning framework that combines semantic and instance segmentation is proposed to extract single road-side trees from vehicle-mounted mobile laser scanning (MLS) point clouds. In the semantic segmentation stage, the ground points are filtered to reduce the processing time. Subsequently, a graph-based semantic segmentation network is developed to segment road-side tree points from the raw MLS point clouds. For the individual tree segmentation stage, a novel joint instance and semantic segmentation network is adopted to detect instance-level roadside trees. Two complex Chinese urban point cloud scenes are used to evaluate the individual urban tree segmentation performance of the proposed method. The proposed method accurately extract approximately 90% of the road-side trees and achieve better segmentation results than existing published methods in both two urban MLS point clouds. Living Vegetation Volume (LVV) calculation can benefit from individual tree segmentation. The proposed method provides a promising solution for ecological construction based on the LVV calculation of urban roads.

Lightweight Semantic Architecture Modeling by 3D Feature Line Detection

Existing architecture semantic modeling methods in 3D complex urban scenes continue facing difficulties, such as limited training data, lack of semantic information, and inflexible model processing. Focusing on extracting and adopting accurate semantic information into a modeling process, this work presents a framework for lightweight modeling of buildings that joints point clouds semantic segmentation and 3D feature line detection constrained by geometric and photometric consistency. The main steps are: (1) Extraction of single buildings from point clouds using 2D-3D semi-supervised semantic segmentation under photometric and geometric constraints. (2) Generation of lightweight building models by using 3D plane-constrained multi-view feature line extraction and optimization. (3) Introduction of detailed semantics of building elements into independent 3D building models by using fine-grained segmentation of multi-view images to achieve high-accuracy architecture lightweight modeling with fine-grained semantic information. Experimental results demonstrate that it can perform independent lightweight modeling of each building on point cloud at various scales and scenes, with accurate geometric appearance details and realistic textures. It also enables independent processing and analysis of each building in the scenario, making them more useful in practical applications.

Optimized Views Photogrammetry: Precision Analysis and a Large-Scale Case Study in Qingdao

Unmanned aerial vehicle (UAVs) have become one of the widely used remote sensing platforms and played a critical role in the construction of smart cities. However, due to the complex environment in urban scenes, secure, and accurate data acquisition brings great challenges to 3-D modeling and scene updating. Optimal trajectory planning of UAVs and accurate data collection of onboard cameras are nontrivial issues in urban modeling. This study presents the principle of optimized views photogrammetry and verifies its precision and potential in large-scale 3-D modeling. Different from oblique photogrammetry, optimized views photogrammetry uses rough models to generate and optimize UAV trajectories, which is achieved through the consideration of model point reconstructability and view point redundancy. Based on the principle of optimized views photogrammetry, this study first conducts a precision analysis of 3-D models by using UAV images of optimized views photogrammetry and then executes a large-scale case study in the urban region of Qingdao City, China, to verify its engineering potential. By using GCPs for image orientation precision analysis and terrestrial laser scanning (TLS) point clouds for model quality analysis, experimental results show that optimized views photogrammetry could construct stable image connection networks and could achieve comparable image orientation accuracy. Benefiting from the accurate image acquisition strategy, the quality of mesh models significantly improves, especially for urban areas with serious occlusions, in which 3 to 5 times of higher accuracy has been achieved. Besides, the case study in Qingdao City verifies that optimized views photogrammetry can be a reliable and powerful solution for the large-scale 3-D modeling in complex urban scenes.

Full Semantic Constructed Network for Urban Use Classification From Very High-Resolution Optical Remote Sensing Imagery

Recently, semantic segmentation technology has been a research hotspot in optical remote sensing urban use classification. However, since coupled semantic relations in very high resolution and complex urban scenes, a more effective semantic description for pixelwise urban use interpretation has become a challenge. Then, aiming to set up a more effective semantic description, the effective receptive field (ERF) is analyzed in general convolutional neural networks, and there is an unreasonable ERF distribution in the stacked convolutional layers of the encoder. This would lead to a large amount of small ERFs and fewer not large enough ERFs to form a naive semantic description at decoder. Therefore, in this article, a novel full semantic constructed network (FSCNet) is proposed to improve the naive semantic description and set up an effective semantic description. First, to avoid noise from shallow feature layers, a residual refinement convolution is designed to optimize the full scale skip connections based on the U-shaped encoder-decoder. Second, an interscale fusion module is newly designed for multiscale feature fusion, which can generate three initial semantic modalities that are prepared for redefining the full semantic description. Third, a multiscale local context spatial attention module and boundary supervision are designed for an initial shallow semantic modality to capture the pure boundary information, and then pyramid spatial pooling is employed for an initial deep semantic modality to further enlarge the ERF and obtain a more abstract global information. Next, a self-calibration convolution combined with the atrous spatial pyramid pooling is designed to rectify and enrich an initial middle semantic modality, which can improve the naive semantic description and bridge the semantic gap between the redefined shallow and deep semantic modalities to advance the full semantic feature fusion. Finally, extensive experiments are carried out on three benchmarks (e.g., ISPRS Vaihinge, Potsdam and DLRSD), and comparative results show that the proposed FSCNet can get remarkable performance compared to state-of-the-art (SOTA) methods. Besides, the code is available at https://github.com/DorisCV/FSCNet.

Mesh-Based DGCNN: Semantic Segmentation of Textured 3-D Urban Scenes

Textured 3D mesh is one of the final user products in photogrammetry and remote sensing. However, research on the semantic segmentation of complex urban scenes represented by textured 3D meshes is in its infancy. We present a mesh-based dynamic graph CNN (DGCNN) for the semantic segmentation of textured 3D meshes. To represent each mesh facet, composite input feature vectors are constructed by concatenating the face-inherent features, i.e., XYZ coordinates of the center of gravity (CoG), texture values, and normal vectors. A texture fusion module is embedded into the proposed mesh-based DGCNN to generate high-level semantic features of the high-resolution texture information, which is useful for semantic segmentation. We achieve competitive accuracies when the proposed method is applied to the SUM mesh datasets. The overall accuracy (OA), Kappa coefficient (Kap), mean precision (mP), mean recall (mR), mean F1 score (mF1), and mean intersection over union (mIoU) are 93.3%, 88.7%, 79.6%, 83.0%, 80.7%, and 69.6%, respectively. In particular, the OA, mean class accuracy (mAcc), mIoU, and mF1 increase by 0.3%, 12.4%, 3.4%, and 6.9%, respectively, compared to the state-of-the-art method.

KD-Tree-Based Euclidean Clustering for Tomographic SAR Point Cloud Extraction and Segmentation

Tomographic Synthetic Aperture Radar (TomoSAR) has three-dimensional imaging capability. Never- theless, its building structure is full of considerable noise and wrong targets owing to the baseline distribution and the algorithm restrictions. Accordingly, efficiently extracting and segmenting buildings from SAR point clouds with huge data is a critical issue. According to the characteristics of building facade point clouds, the KD-Tree-based Euclidean clustering is introduced as a fast-clustering method for better extracting and segmenting individual building facades. At first, a simple morphological filter (SMRF) is utilized to identify the ground and non-ground point clouds. The building facades are then extracted from the non-ground point clouds based on density information. Subsequently, the KD-Tree-based Euclidean clustering method denoises the building facades and segments them into individual facades. Finally, a second Euclidean clustering is performed for each of these facades to denoise them again. Experimental results indicate that the presented approach can prepare accurate and efficient extraction and segmentation results for complex urban scenes.

Automatic registration of optical image and airborne LiDAR data based on centers and corner points of building boundaries

ABSTRACT Applications based on the fusion of aerial imagery and Light Detection and Ranging (LiDAR) data have attracted more and more interest in recent years. Registration is the crucial prerequisite of data integration because of the spatial shift between two data modalities. However, there are many challenges for registration due to the different mechanisms of LiDAR and camera, especially in complex urban scenes. To address the problem, this paper proposes an automatic and robust registration method by using the centres and corner points of building boundaries as tie points. Primary registration is achieved by using centers of building boundaries as control points. To increase the robustness and accuracy of the registration, corner points of building boundaries are utilized to estimate the transformation model repeatedly. Two datasets of optical imagery and LiDAR data with different scene coverage are used to verify the performance of the proposed method. Experimental results show that the proposed method can achieve the registration accuracy of less than 2 pixels for both cases based on an assessment of check lines. This result can basically meet the requirements of data fusion.

UNCERTAINTY REPRESENTATION AND QUANTIFICATION OF 3D BUILDING MODELS

Abstract. The quality of environmental perception is of great interest for localization tasks in autonomous systems. Maps, generated from the sensed information, are often used as additional spatial references in these applications. The quantification of the map uncertainties gives an insight into how reliable and complete the map is, avoiding the potential systematic deviation in pose estimation. Mapping 3D buildings in urban areas using Light detection and ranging (LiDAR) point clouds is a challenging task as it is often subject to uncertain error sources in the real world such as sensor noise and occlusions, which should be well represented in the 3D models for the downstream localization tasks. In this paper, we propose a method to model 3D building façades in complex urban scenes with uncertainty quantification, where the uncertainties of windows and façades are indicated in a probabilistic fashion. The potential locations of the missing objects (here: windows) are inferred by the available data and layout patterns with the Monte Carlo (MC) sampling approach. The proposed 3D building model and uncertainty measures are evaluated using the real-world LiDAR point clouds collected by Riegl Mobile Mapping System. The experimental results show that our uncertainty representation conveys the quality information of the estimated locations and shapes for the modelled map objects.

Measuring PM2.5 Concentrations from a Single Smartphone Photograph

PM2.5 participates in light scattering, leading to degraded outdoor views, which forms the basis for estimating PM2.5 from photographs. This paper devises an algorithm to estimate PM2.5 concentrations by extracting visual cues and atmospheric indices from a single photograph. While air quality measurements in the context of complex urban scenes are particularly challenging, when only a single atmospheric index or cue is given, each one can reinforce others to yield a more robust estimator. Therefore, we selected an appropriate atmospheric index in various outdoor scenes to identify reasonable cue combinations for measuring PM2.5. A PM2.5 dataset (PhotoPM-daytime) was built and used to evaluate performance and validate efficacy of cue combinations. Furthermore, a city-wide experiment was conducted using photographs crawled from the Internet to demonstrate the applicability of the algorithm in large-area PM2.5 monitoring. Results show that smartphones equipped with the developed method could potentially be used as PM2.5 sensors.

A Self-Attentive Hybrid Coding Network for 3D Change Detection in High-Resolution Optical Stereo Images

Real-time monitoring of urban building development provides a basis for urban planning and management. Remote sensing change detection is a key technology for achieving this goal. Intelligent change detection based on deep learning of remote sensing images is a current focus of research. However, most methods only use unimodal remote sensing data and ignore vertical features, leading to incomplete characterization, poor detection of small targets, and false detections and omissions. To solve these problems, we propose a multi-path self-attentive hybrid coding network model (MAHNet) that fuses high-resolution remote sensing images and digital surface models (DSMs) for 3D change detection of urban buildings. We use stereo images from the Gaofen-7 (GF-7) stereo mapping satellite as the data source. In the encoding stage, we propose a multi-path hybrid encoder, which is a structure that can efficiently perform multi-dimensional feature mining of multimodal data. In the deep feature fusion link, a dual self-attentive fusion structure is designed that can improve the deep feature fusion and characterization of multimodal data. In the decoding stage, a dense skip-connection decoder is designed that can fuse multi-scale features flexibly and reduce spatial information losses in small-change regions in the down-sampling process, while enhancing feature utilization and propagation efficiency. Experimental results show that MAHNet achieves accurate pixel-level change detection in complex urban scenes with an overall accuracy of 97.44% and F1-score of 92.59%, thereby outperforming other methods of change detection.

Joint Trajectory Design and Resource Allocation for IRS-Assisted UAV Communications With Wireless Energy Harvesting

In this letter, we present a novel intelligent reflecting surface (IRS)-assisted unmanned aerial vehicle (UAV) communication system, where both the IRS and ground users (GUs) can achieve self-sustainable operations by exploiting the energy harvesting (EH) technique. We consider a practical scenario that the direct links between the UAV and GUs always suffer from blockages, typically in complex urban scene. To mitigate this adverse effect, the IRS can be properly deployed to create a clear propagation environment. We maximize the sum-rate for the proposed system by jointly optimizing the IRS’s phase-shift, transmit power and time allocation of GUs, and trajectory of the UAV. To handle the non-convexity caused by coupled variables, the original problem is decomposed into three subproblems and solved by the block coordinate descent (BCD) method. Through numerical results, the proposed algorithm can achieve significant performance gains over the benchmarks.