Urban Scene Classification Research Articles

Multi-scale multi-temporal landscape pattern analysis using high spatial urban scene classification

ABSTRACTUrban landscape pattern analysis is conducive to understanding the land cover distribution and changes and promoting urban spatial layout rationality and sustainable development. Previous research has often used low and medium spatial resolution data to classify urban land cover at the pixel level and then analyse landscape pattern indices. However, the underlying semantic information provided by pixel classification needs to consider the spatial correlation of surface features, and it is challenging to obtain high-level semantic information on urban functional areas and analyse the relationship between human activities and land use. Leveraging the abundant spatial characteristics in high-resolution remote sensing images, high spatial urban scene classification can infer the feature categories and spatial relationships between features to realize higher-level image interpretation and obtain regional urban scene details. And it contributes to urban planning and decision-making, better meeting the different demands. Therefore, this study aims at a novel methodological framework for landscape pattern analysis of cross-temporal multi-scale urban scene classification. The results imply that: (1) The high-resolution images can make up for the lack of information on urban functional zones in low and medium resolution images; (2) Landscape pattern analysis based on scene classification can analyse urban spatial relationships more rationally than traditional research; and (3) With the continuous advancement of urbanization, the urban functional zoning becomes more apparent, and the dominant industries have changed significantly from industry to commerce and services. This study expands the analytical application range of landscape ecology and remote sensing technology and provides new inspirations for urban planning, environmental monitoring, ecological protection.

Hyperspectral image-aided LiDAR point cloud labeling via spatio-spectral feature representation learning

Urban scene-level 3D point cloud labeling is a very laborious and expensive task compared to images. Conversely however, image processing techniques, deep learning or otherwise are more established and mature. Thus, in a multi-source data environment, the labeling of a point cloud scene via an automated image process as an initial step, followed by a manual human verification process is an effective way to save man hours and cost. With the above as the goal, this study presents a simple but robust spatio-spectral feature representation approach. In this approach, a class-aware band selection and reduction (CBSR) technique is developed for optimal hyperspectral feature representation. A double-branched convolutional Gaussian Bernoulli deep belief network (CGBDBN) is then used for hierarchical spatial feature extraction from LiDAR-derived data and the CBSR data. Using stacked ensemble learning, spatio-spectral features are generated from the two feature streams via a fusion rule and then classified — the results of which are used in labeling a raw 3D LiDAR point cloud through projection. To evaluate this study, extensive experiments were conducted on the IEEE 2018 Houston dataset — the only publicly available dataset with both hyperspectral image (HSI) and 3D point cloud covering the same area — for urban scene classification. The results indicate that the developed CBSR attained comparatively competitive results with state-of-the-art approaches, thus making it a robust spectral feature representation technique. Also, the weight-sharing property, probabilistic modeling, and hierarchical nature of CGBDBN gives our approach the ability to capture high-level contextual features. Furthermore, compared to the spatial- or spectral-only features, the generated spatio-spectral features are more discriminative and significantly aided in improving the proposed model’s efficacy. Overall, the proposed approach, based on the evaluation metrics, is a robust and effective approach for both coarse- and fine-grained raw LiDAR point cloud labeling tasks.

Resolution invariant urban scene classification using Multiview learning paradigm

Urban scene classification is an interesting area in computer vision. The task involves classifying a scene from a pair of aerial-view and ground-view images. Existing approaches have considered single-view and Multiview methods. Multiview approaches have shown to be more robust than single-view. However, most existing Multiview approaches neglect the disparity in the resolution of both views. The aerial-view images are captured with sophisticated high-resolution remote sensing devices. While the ground-view images are captured from closer perspectives with lower resolution. This paper proposed a Multiview scene classification (MuSC) model that caters to the resolution disparity in both views. MuSC introduces a Fourier convolution network (FCN) that is robust to variation of resolution in the cross-view images. The FCN is designed to extract local features (in spatial domain) and global features (in spectral domain). The proposed MuSC has a two-stage classifier. The first stage trains a discriminative view-specific network and classifies each view separately. However, the outputs from each view-specific network are projected into a unified subspace and mutual agreement between them is incentivized through contrastive learning. The second stage integrates the predictions from each view-specific network and trains a unified classifier for final prediction. This integration encourages cross-view complementarity. MuSC is evaluated on two datasets, AiRound and CV-BrCT. Several experiments are conducted with different settings. The results demonstrate that MuSC outperforms existing state-of-the-art models.

Classification of Complicated Urban Forest Acoustic Scenes with Deep Learning Models

The use of passive acoustic monitoring (PAM) can compensate for the shortcomings of traditional survey methods on spatial and temporal scales and achieve all-weather and wide-scale assessment and prediction of environmental dynamics. Assessing the impact of human activities on biodiversity by analyzing the characteristics of acoustic scenes in the environment is a frontier hotspot in urban forestry. However, with the accumulation of monitoring data, the selection and parameter setting of the deep learning model greatly affect the content and efficiency of sound scene classification. This study compared and evaluated the performance of different deep learning models for acoustic scene classification based on the recorded sound data from Guangzhou urban forest. There are seven categories of acoustic scenes for classification: human sound, insect sound, bird sound, bird–human sound, insect–human sound, bird–insect sound, and silence. A dataset containing seven acoustic scenes was constructed, with 1000 samples for each scene. The requirements of the deep learning models on the training data volume and training epochs in the acoustic scene classification were evaluated through several sets of comparison experiments, and it was found that the models were able to achieve satisfactory accuracy when the training sample data volume for a single category was 600 and the training epochs were 100. To evaluate the generalization performance of different models to new data, a small test dataset was constructed, and multiple trained models were used to make predictions on the test dataset. All experimental results showed that the DenseNet_BC_34 model performs best among the comparison models, with an overall accuracy of 93.81% for the seven acoustic scenes on the validation dataset. This study provides practical experience for the application of deep learning techniques in urban sound monitoring and provides new perspectives and technical support for further exploring the relationship between human activities and biodiversity.

Systematic Comparison of Objects Classification Methods Based on ALS and Optical Remote Sensing Images in Urban Areas

Geographical object classification and information extraction is an important topic for the construction of 3D virtual reality and digital twin cities in urban areas. However, the majority of current multi-target classification of urban scenes uses only a single source data (optical remote sensing images or airborne laser scanning (ALS) point clouds), which is limited by the restricted information of the data source itself. In order to make full use of the information carried by multiple data sources, we often need to set more parameters as well as algorithmic steps. To address the above issues, we compared and analyzed the object classification methods based on data fusion of airborne LiDAR point clouds and optical remote sensing images, systematically. Firstly, the features were extracted and determined from airborne LiDAR point clouds and high-resolution optical images. Then, some key feature sets were selected and were composed of median absolute deviation of elevation, normalized elevation values, texture features, normal vectors, etc. The feature sets were fed into various classifiers, such as random forest (RF), decision tree (DT), and support vector machines (SVM). Thirdly, the suitable feature sets with appropriate dimensionality were composed, and the point clouds were classified into four categories, such as trees (Tr), houses and buildings (Ho), low-growing vegetation (Gr), and impervious surfaces (Is). Finally, the single data source and multiple data sources, the crucial feature sets and their roles, and the resultant accuracy of different classifier models were compared and analyzed. Under the conditions of different experimental regions, sampling proportion parameters and machine learning models, the results showed that: (1) the overall classification accuracy obtained by the feature-level data fusion method was 76.2% compared with the results of only a single data source, which could improve the overall classification accuracy by more than 2%; (2) the accuracy of the four classes in the urban scenes can reach 88.5% (Is), 76.7% (Gr), 87.2% (Tr), and 88.3% (Ho), respectively, while the overall classification accuracy can reach 87.6% with optimal sampling parameters and random forest classifiers; (3) the RF classifier outperforms DT and SVM for the same sample conditions. In this paper, the method based on ALS point clouds and image data fusion can accurately classify multiple targets in urban scenes, which can provide technical support for 3D scene reconstruction and digital twin cities in complex geospatial environments.

Using different training strategies for urban land-use classification based on convolutional neural networks

Urban land-use scene classification from high-resolution remote-sensing imagery at high quality and accuracy is of paramount interest for urban planning, government policy-making and urban change detection. In recent years, urban land-use classification has become an ongoing task in areas addressable primarily by remote sensing, and numerous deep learning algorithms have achieved high performance on this task. However, both dataset and methodology problems still exist in the current approaches. Previous studies have relied on limited data sources, resulting in saturated classification results, and they have difficulty achieving comprehensive classification results. The previous methods based on convolutional neural networks (CNNs) focused primarily on model architecture rather than on the hyperparameters. Therefore, to achieve more accurate classification results, in this study, we constructed a new large dataset for urban land-use scene classification. More than thirty thousand remote sensing scene images were collected to create a dataset with balanced class samples that includes both higher intra-class variations and smaller inter-class dissimilarities than do the previously available public datasets. Then, we analysed two possible strategies for exploiting the capabilities of three existing popular CNNs on our datasets: full training and fine tuning. For each strategy, three types of learning rate decay were applied: fixed, exponential and polynomial. The experimental results indicate that fine tuning tends to be the best-performing strategy, and using ResNet-V1-50 and polynomial learning rate decay achieves the best results for the urban land-use scene classification task.

Application of a graph convolutional network with visual and semantic features to classify urban scenes

Urban scenes consist of visual and semantic features and exhibit spatial relationships among land-use types (e.g. industrial areas are far away from the residential zones). This study applied a graph convolutional network with neighborhood information (henceforth, named the neighbour supporting graph convolutional neural network), to learn spatial relationships for urban scene classification. Furthermore, a co-occurrence analysis with visual and semantic features proceeded to improve the accuracy of urban scene classification. We tested the proposed method with the fifth ring road of Beijing with an overall classification accuracy of 0.827 and a Kappa coefficient of 0.769. In comparison with other methods, such as support vector machine, random forest, and general graph convolutional network, the case study showed that the proposed method improved about 10% in urban scene classification.

Multi-View Urban Scene Classification with a Complementary-Information Learning Model

Traditional urban scene-classification approaches focus on images taken either by satellite or in aerial view. Although single-view images are able to achieve satisfactory results for scene classification in most situations, the complementary information provided by other image views is needed to further improve performance. Therefore, we present a complementary information-learning model (CILM) to perform multi-view scene classification of aerial and ground-level images. Specifically, the proposed CILM takes aerial and ground-level image pairs as input to learn view-specific features for later fusion to integrate the complementary information. To train CILM, a unified loss consisting of cross entropy and contrastive losses is exploited to force the network to be more robust. Once CILM is trained, the features of each view are extracted via the two proposed feature-extraction scenarios and then fused to train the support vector machine classifier for classification. The experimental results on two publicly available benchmark data sets demonstrate that CILM achieves remarkable performance, indicating that it is an effective model for learning complementary information and thus improving urban scene classification.

Dynamic sensor activation and decision-level fusion in wireless acoustic sensor networks for classification of domestic activities

For the past decades there has been a rising interest for wireless sensor networks to obtain information about an environment. One interesting modality is that of audio, as it is highly informative for numerous applications including speech recognition, urban scene classification, city monitoring, machine listening and classifying domestic activities. However, as they operate at prohibitively high energy consumption, commercialisation of battery-powered wireless acoustic sensor networks has been limited. To increase the network’s lifetime, this paper explores the joint use of decision-level fusion and dynamic sensor activation. Hereby adopting a topology where processing – including feature extraction and classification – is performed on a dynamic set of sensor nodes that communicate classification outputs which are fused centrally. The main contribution of this paper is the comparison of decision-level fusion with different dynamic sensor activation strategies on the use case of automatically classifying domestic activities. Results indicate that using vector quantisation to encode the classification output, computed at each sensor node, can reduce the communication per classification output to 8 bit without loss of significant performance. As the cost for communication is reduced, local processing tends to dominate the overall energy budget. It is indicated that dynamic sensor activation, using a centralised approach, can reduce the average time a sensor node is active up to 20% by leveraging redundant information in the network. In terms of energy consumption, this resulted in an energy reduction of up to 80% as the cost for computation dominates the overall energy budget.

Estimating Proportion of Vegetation Cover at the Vicinity of Archaeological Sites Using Sentinel-1 and -2 Data, Supplemented by Crowdsourced OpenStreetMap Geodata

Monitoring vegetation cover is an essential parameter for assessing various natural and anthropogenic hazards that occur at the vicinity of archaeological sites and landscapes. In this study, we used free and open access to Copernicus Earth Observation datasets. In particular, the proportion of vegetation cover is estimated from the analysis of Sentinel-1 radar and Sentinel-2 optical images, upon their radiometric and geometric corrections. Here, the proportion of vegetation based on the Radar Vegetation Index and the Normalized Difference Vegetation Index is estimated. Due to the medium resolution of these datasets (10 m resolution), the crowdsourced OpenStreetMap service was used to identify fully and non-vegetated pixels. The case study is focused on the western part of Cyprus, whereas various open-air archaeological sites exist, such as the archaeological site of “Nea Paphos” and the “Tombs of the Kings”. A cross-comparison of the results between the optical and the radar images is presented, as well as a comparison with ready products derived from the Sentinel Hub service such as the Sentinel-1 Synthetic Aperture Radar Urban and Sentinel-2 Scene classification data. Moreover, the proportion of vegetation cover was evaluated with Google Earth red-green-blue free high-resolution optical images, indicating that a good correlation between the RVI and NDVI can be generated only over vegetated areas. The overall findings indicate that Sentinel-1 and -2 indices can provide a similar pattern only over vegetated areas, which can be further elaborated to estimate temporal changes using integrated optical and radar Sentinel data. This study can support future investigations related to hazard analysis based on the combined use of optical and radar sensors, especially in areas with high cloud-coverage.

Semantic and Spatial Co-Occurrence Analysis on Object Pairs for Urban Scene Classification

Urban scene classifications with very-high-resolution (VHR) satellite images play an important role in urban functional-zone investigation and landscape analysis. Most scene classification techniques utilize visual features which however are weak in identifying scenes, as scenes are usually composed of diverse objects with variant visual cues. To resolve this issue, we use spatial object relations to characterize urban scenes and recognize their categories. Accordingly, two questions are considered in this study: how to measure spatial object relations and how to use these relations to classify urban scenes? First, we propose a novel scene feature, semantic and spatial co-occurrence probability (SSCP), to measure spatial relations between objects with considering their directions, distances, and semantics, which can hopefully resolve the three key issues in measuring spatial object relations, i.e., anisotropy, scale- and semantic-dependencies. Then, a semilatent Dirichlet allocation is employed to classify scenes based on the proposed SSCP. In experiments, our method is first verified based on the UC Merced data set, and then used to generate an urban functional-zone map for Beijing, China. Both experiments indicate high accuracies and state-of-the-art performances of our method. In addition, we give a deep insight into SSCP's characteristics including convergence, high efficiency, and invariance to affine transformation, which can make our method more applicable.

Angular difference feature extraction for urban scene classification using ZY-3 multi-angle high-resolution satellite imagery

Spaceborne multi-angle images with a high-resolution are capable of simultaneously providing spatial details and three-dimensional (3D) information to support detailed and accurate classification of complex urban scenes. In recent years, satellite-derived digital surface models (DSMs) have been increasingly utilized to provide height information to complement spectral properties for urban classification. However, in such a way, the multi-angle information is not effectively exploited, which is mainly due to the errors and difficulties of the multi-view image matching and the inaccuracy of the generated DSM over complex and dense urban scenes. Therefore, it is still a challenging task to effectively exploit the available angular information from high-resolution multi-angle images. In this paper, we investigate the potential for classifying urban scenes based on local angular properties characterized from high-resolution ZY-3 multi-view images. Specifically, three categories of angular difference features (ADFs) are proposed to describe the angular information at three levels (i.e., pixel, feature, and label levels): (1) ADF-pixel: the angular information is directly extrapolated by pixel comparison between the multi-angle images; (2) ADF-feature: the angular differences are described in the feature domains by comparing the differences between the multi-angle spatial features (e.g., morphological attribute profiles (APs)). (3) ADF-label: label-level angular features are proposed based on a group of urban primitives (e.g., buildings and shadows), in order to describe the specific angular information related to the types of primitive classes. In addition, we utilize spatial-contextual information to refine the multi-level ADF features using superpixel segmentation, for the purpose of alleviating the effects of salt-and-pepper noise and representing the main angular characteristics within a local area. The experiments on ZY-3 multi-angle images confirm that the proposed ADF features can effectively improve the accuracy of urban scene classification, with a significant increase in overall accuracy (3.8–11.7%) compared to using the spectral bands alone. Furthermore, the results indicated the superiority of the proposed ADFs in distinguishing between the spectrally similar and complex man-made classes, including roads and various types of buildings (e.g., high buildings, urban villages, and residential apartments).

Improved Urban Scene Classification Using Full-Waveform Lidar

Full-waveform lidar data provides supplementary radiometric as well as more accurate geometric target information, when compared to discrete return systems. In this research, a wide range of classes in an urban scene; including trees, medium vegetation, low vegetation (grass), water bodies, pitched roofs, flat roofs, asphalt, vehicles, power lines, walls (fences) and concrete are considered. In order to tackle the challenge of distinguishing geometrically similar classes and enhancing the separability of other targets, a new set of features based on deconvolved waveforms is introduced. The positive effect of the proposed feature dataset on classification accuracy in individual classes is shown using two ensemble classifiers (random forests and RUSBoost). Performance of the classifiers is improved by integration with sampling techniques, especially for the under-represented classes. The final output of the proposed method is a highly detailed land cover map of the urban scene, which affords good separability between critical classes.

Enhanced Autocorrelation-Based Algorithms for Filtering Airborne Lidar Data over Urban Areas

AbstractMany existing algorithms for light detection and ranging (lidar) data classification are known to perform reliably; however, the automation of the classification of complex urban scenes is still a challenging problem. In this paper, two classification algorithms based on spatial autocorrelation statistics, such as the Local Moran’s I and the Getis-Ord Gi*, are proposed. These autocorrelation statistics are computed over sample urban areas, including complex terrain with diverse building characteristics. The proposed autocorrelation-based algorithms are applied to airborne lidar point clouds over the complex urban areas to generate highly accurate digital elevation models (DEMs) and classify the lidar points as ground and nonground points by using the DEMs. It is also demonstrated that the minimum-based rasterization and slope-based filtering can be integrated to effectively remove outliers from the DEMs. The test results showed that the autocorrelation-based algorithms produce high-level assessmen...

Semantic Classification of Heterogeneous Urban Scenes Using Intrascene Feature Similarity and Interscene Semantic Dependency

Semantic classification of urban scenes aims to classify scenes composed of many different types of objects into predefined semantic classes. To learn the association between urban scenes and semantic classes, five tasks are needed: 1) segmenting the image into scenes; 2) establishing semantic classes of scenes; 3) extracting and transforming features; 4) measuring the intrascenes feature similarity; and 5) labeling each scene by a semantic classification method. Despite many efforts on these tasks, most existing works consider only visual features with inconsistent similarity measurement, while ignore semantic features inside scenes and the interactions between scenes, leading to poor classification results for high heterogeneous scenes. To solve these problems, this study combines intrascene feature similarity and interscene semantic dependency to form a two-step classification approach. For the first step, visual and semantic features are first optimized to be invariant to affine transformation, and then are employed in K-Nearest Neighbor to initially classify scenes. For the second step, multinomial distribution is presented to model both the spatial and semantic dependency between scenes, and then used to improve the initial classification results. The implementations conducted in two study areas indicate that the proposed approach produces better results for heterogeneous scenes than visual interpretation, as it can discover and model the hidden information between scenes which is often ignored by existing methods. In addition, compared with the initial classification, the optimized step improves accuracies by 3.6% and 5% in the two study areas, respectively.

Geometry and Colour Based Classification of Urban Point Cloud Scenes Using a Supervised Self-Organizing Map

Geometry and Colour Based Classification of Urban Point Cloud Scenes Using a Supervised Self-Organizing Map

Point cloud segmentation for urban scene classification

Abstract. High density point clouds of urban scenes are used to identify object classes like buildings, vegetation, vehicles, ground, and water. Point cloud segmentation can support classification and further feature extraction provided that the segments are logical groups of points belonging to the same object class. A single segmentation method will typically not provide a satisfactory segmentation for a variety of classes. This paper explores the combination of various segmentation and post-processing methods to arrive at useful point cloud segmentations. A feature based on the normal vector and flatness of a point neighbourhood is used to group cluttered points in trees as well as points on surfaces in areas where the extraction of planes was not successful. Combined with segment merging and majority filtering large segments can be obtained allowing the derivation of accurate segment feature values. Results are presented and discussed for a 70 million point dataset over a part of Rotterdam.

Super-Segments Based Classification of 3D Urban Street Scenes

We address the problem of classifying 3D point clouds: given 3D urban street scenes gathered by a lidar sensor, we wish to assign a class label to every point. This work is a key step toward realizing applications in robots and cars, for example. In this paper, we present a novel approach to the classification of 3D urban scenes based on super-segments, which are generated from point clouds by two stages of segmentation: a clustering stage and a grouping stage. Then, six effective normal and dimension features that vary with object class are extracted at the super-segment level for training some general classifiers. We evaluate our method both quantitatively and qualitatively using the challenging Velodyne lidar data set. The results show that by only using normal and dimension features we can achieve better recognition than can be achieved with high-dimensional shape descriptors. We also evaluate the adopting of the MRF framework in our approach, but the experimental results indicate that thisbarely improved the accuracy of the classified results due to the sparse property of the super-segments.

Combining pyramid representation and AdaBoost for urban scene classification using high-resolution synthetic aperture radar images

This study presents a new algorithm called pyramid representation (PR)-AdaBoost, which combines PR and AdaBoost for urban area classification using high-resolution synthetic aperture radar (SAR) images. PR is used to hierarchically represent local feature sets and AdaBoost is used to choose proper features from the PR vector and effectively discriminate categories. The authors evaluate the proposed algorithm on a data set consisting of high-resolution SAR images of five different categories of scene and on a real TerraSAR-X image. The experimental results have shown that PR-AdaBoost can achieve higher classification accuracy than AdaBoost based on global representation or local representation such as bag-of-features. It also outperforms classical classifiers such as nearest neighbour, boosted distance and support vector machine based on the same representation.

Urban Scene Classification Based on Multi-dimensional Pyramid Representation and AdaBoost Using High Resolution SAR Images

This paper presents a new image representation algorithm called multi-dimensional pyramid representation (MPR),and combines MPR and AdaBoost (MPR-AdaBoost) for urban scene classification using high resolution SAR images.MPR calculates a pyramid representation (PR) vector in each dimension of local feature and combines the PR vectors together to get an MPR vector.The computational complexity of PR is high when the local feature is high-dimensional,which leads PR vector to lose discriminative information in real applications.MPR overcomes these limitations.Its computational complexity is low and the MPR vector has discriminative information,even when the local feature is high-dimensional.Using a TerraSAR-X data set and a TerraSAR-X image,AdaBoost based on PR (PR-AdaBoost) and MPR-AdaBoost are compared.The experimental results have shown that MPR-AdaBoost gives comparable results and reduces the computational cost.