Images Of Urban Scenes Research Articles

Multi-Resolution Learning and Semantic Edge Enhancement for Super-Resolution Semantic Segmentation of Urban Scene Images.

Super-resolution semantic segmentation (SRSS) is a technique that aims to obtain high-resolution semantic segmentation results based on resolution-reduced input images. SRSS can significantly reduce computational cost and enable efficient, high-resolution semantic segmentation on mobile devices with limited resources. Some of the existing methods require modifications of the original semantic segmentation network structure or add additional and complicated processing modules, which limits the flexibility of actual deployment. Furthermore, the lack of detailed information in the low-resolution input image renders existing methods susceptible to misdetection at the semantic edges. To address the above problems, we propose a simple but effective framework called multi-resolution learning and semantic edge enhancement-based super-resolution semantic segmentation (MS-SRSS) which can be applied to any existing encoder-decoder based semantic segmentation network. Specifically, a multi-resolution learning mechanism (MRL) is proposed that enables the feature encoder of the semantic segmentation network to improve its feature extraction ability. Furthermore, we introduce a semantic edge enhancement loss (SEE) to alleviate the false detection at the semantic edges. We conduct extensive experiments on the three challenging benchmarks, Cityscapes, Pascal Context, and Pascal VOC 2012, to verify the effectiveness of our proposed MS-SRSS method. The experimental results show that, compared with the existing methods, our method can obtain the new state-of-the-art semantic segmentation performance.

Efficient multi-scale network for semantic segmentation of fine-resolution remotely sensed images

Abstract Semantic segmentation of remote sensing urban scene images has diverse practical applications, including land cover mapping, urban change detection, environmental protection, and economic evaluation. However, classical semantic segmentation networks encounter challenges such as inadequate utilization of multi-scale semantic information and imprecise edge target segmentation in high-resolution remote sensing images. In response, this article introduces an efficient multi-scale network (EMNet) tailored for semantic segmentation of common features in remote sensing images. To address these challenges, EMNet integrates several key components. Firstly, the efficient atrous spatial pyramid pooling module is employed to enhance the relevance of multi-scale targets, facilitating improved extraction and processing of context information across different scales. Secondly, the efficient multi-scale attention mechanism and multi-scale jump connections are utilized to fuse semantic features from various levels, thereby achieving precise segmentation boundaries and accurate position information. Finally, an encoder-decoder structure is incorporated to refine the segmentation results. The effectiveness of the proposed network is validated through experiments conducted on the publicly available DroneDeploy image dataset and Potsdam dataset. Results indicate that EMNet achieves impressive performance metrics, with mean intersection over union (MIoU), mean precision (MPrecision), and mean recall (MRecall) reaching 75.99%, 86.76%, and 85.07%, respectively. Comparative analysis demonstrates that the network proposed in this article outperforms current mainstream semantic segmentation networks on both the DroneDeploy and Potsdam dataset.

Quantifying urban environments: Aesthetic preference through the lens of prospect-refuge theory

Prospect-refuge theory suggests that people prefer environments that offer both prospect, the ability to scan for resources, and refuge, a safe place to hide. Urban planners, architects and researchers alike have had a tendency to use prospect-refuge theory research on natural scenes to inform on the design of urban environments. Despite the large body of prospect-refuge theory research, the degree to which prospect and refuge impact preference in urban environments remain unclear. Here, we aim to first evaluate the relationship between prospect, refuge and preference for urban scene images. Secondly, we aim to evaluate the contributions of visual features and streetscape quality ratings to subjective ratings of prospect and refuge in order to create proxy values of prospect and refuge. Finally, we aim to understand how the proxy values impact preference for urban scenes, and if the proxy values created replicate the relationship between subjective measures of prospect, refuge and preference. First, we used participant ratings of prospect and refuge to predict participants' preference for 552 images of urban street scenes. Higher ratings of both prospect and refuge predicted greater image preference. We next used principal components analysis to summarize these images' low- and high-level visual features as well as participant ratings of streetscape qualities, such as walkability and disorder. Visual feature and streetscape quality principal components predicted prospect and refuge ratings in this first image set, providing “proxy measures' for prospect and refuge. In an independent set of 1119 images from Talen et al. (2022) for which prospect and refuge ratings were not available, we asked whether these proxies for prospect and refuge predicted preference. Findings replicated the effect that more refuge in an image predicts more preference. However, the proxy measure of prospect did not predict preference. In summary, our results show that refuge ratings do relate to preferences in urban environments, which extends prospect-refuge theory to more urban environments. Future work is needed to understand if prospect has different implications in more urban environments.

Segmenting Urban Scene Imagery in Real Time Using an Efficient UNet-like Transformer

Semantic segmentation of high-resolution remote sensing urban images is widely used in many fields, such as environmental protection, urban management, and sustainable development. For many years, convolutional neural networks (CNNs) have been a prevalent method in the field, but the convolution operations are deficient in modeling global information due to their local nature. In recent years, the Transformer-based methods have demonstrated their advantages in many domains due to the powerful ability to model global information, such as semantic segmentation, instance segmentation, and object detection. Despite the above advantages, Transformer-based architectures tend to incur significant computational costs, limiting the model’s real-time application potential. To address this problem, we propose a U-shaped network with Transformer as the decoder and CNN as the encoder to segment remote sensing urban scene images. For efficient segmentation, we design a window-based, multi-head, focused linear self-attention (WMFSA) mechanism and further propose the global–local information modeling module (GLIM), which can capture both global and local contexts through a dual-branch structure. Experimenting on four challenging datasets, we demonstrate that our model not only achieves a higher segmentation accuracy compared with other methods but also can obtain competitive speeds to enhance the model’s real-time application potential. Specifically, the mIoU of our method is 68.2% and 52.8% on the UAVid and LoveDA datasets, respectively, while the speed is 114 FPS, with a 1024 × 1024 input on a single 3090 GPU.

BFANet: Effective segmentation network for low altitude high-resolution urban scene image

The semantic segmentation of low altitude high-resolution urban scene images taken by UAV plays an important role in city management. However, such images have the characteristics of inter-class homogeneity and intra-class heterogeneity. How to segment these images quickly and accurately is still challenging. In this paper, we propose a novel double-branch network. For the challenge of inter-class homogeneity, a boundary flow module is designed to enhance the flow of latent semantic information between two branches by imposing boundary constraints between classes. To alleviate intra-class heterogeneity, a context extraction module based on adaptive dynamic fusion is designed, which effectively captures the long-term relationship of features with very low parameters. Experiments on two typical datasets show that our approach achieves the best balance between accuracy and speed. Specifically, we achieve 65.8% mIoU and 74.1% mIoU on UAVid test set and UDD validation set respectively, and 60FPS on an NVIDIA TITAN Xp.

Enhancing Efficient Global Understanding Network With CSWin Transformer for Urban Scene Images Segmentation

Enhancing Efficient Global Understanding Network With CSWin Transformer for Urban Scene Images Segmentation

CTMFNet: CNN and Transformer Multiscale Fusion Network of Remote Sensing Urban Scene Imagery

Semantic segmentation of remotely sensed urban scene images is widely demanded in areas such as land cover mapping, urban change detection, and environmental protection. With the development of deep learning, methods based on convolutional neural networks (CNNs) have been dominant due to their powerful ability to represent hierarchical feature information. However, the limitations of the convolution operation itself limit the network’s ability to extract global contextual information. With the successful use of transformer in computer vision in recent years, transformer has shown great potential for modeling global contextual information. However, transformer is not sufficiently capable of capturing local detailed information. In this article, to explore the potential of the joint CNN and transformer mechanism for semantic segmentation of remotely sensed urban scenes, we propose a CNN and transformer multiscale fusion network (CTMFNet) based on encoding–decoding for urban scene understanding. To couple local–global context information more efficiently, we designed a dual backbone attention fusion module (DAFM) to couple the local and global context information of the dual-branch encoder. In addition, to bridge the semantic gap between scales, we built a multi-layer dense connectivity network (MDCN) as our decoder. The MDCN enables the full flow of semantic information between multiple scales to be fused with each other through upsampling and residual connectivity. We conducted extensive subjective and objective comparison experiments and ablation experiments on both the International Society of Photogrammetry and Remote Sensing (ISPRS) Vaihingen and ISPRS Potsdam datasets. Numerous experimental results have proven the superiority of our method compared to currently popular methods.

UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery

Semantic segmentation of remotely sensed urban scene images is required in a wide range of practical applications, such as land cover mapping, urban change detection, environmental protection, and economic assessment. Driven by rapid developments in deep learning technologies, the convolutional neural network (CNN) has dominated semantic segmentation for many years. CNN adopts hierarchical feature representation, demonstrating strong capabilities for information extraction. However, the local property of the convolution layer limits the network from capturing the global context. Recently, as a hot topic in the domain of computer vision, Transformer has demonstrated its great potential in global information modelling, boosting many vision-related tasks such as image classification, object detection, and particularly semantic segmentation. In this paper, we propose a Transformer-based decoder and construct an UNet-like Transformer (UNetFormer) for real-time urban scene segmentation. For efficient segmentation, the UNetFormer selects the lightweight ResNet18 as the encoder and develops an efficient global–local attention mechanism to model both global and local information in the decoder. Extensive experiments reveal that our method not only runs faster but also produces higher accuracy compared with state-of-the-art lightweight models. Specifically, the proposed UNetFormer achieved 67.8% and 52.4% mIoU on the UAVid and LoveDA datasets, respectively, while the inference speed can achieve up to 322.4 FPS with a 512 × 512 input on a single NVIDIA GTX 3090 GPU. In further exploration, the proposed Transformer-based decoder combined with a Swin Transformer encoder also achieves the state-of-the-art result (91.3% F1 and 84.1% mIoU) on the Vaihingen dataset. The source code will be freely available at https://github.com/WangLibo1995/GeoSeg.

A Change of Scenery: Does Exposure to Images of Nature Affect Delay Discounting and Food Desirability?

Previous research suggests that exposure to nature may reduce delay discounting (the tendency to discount larger future gains in favor of smaller immediate rewards) and thereby facilitate healthier dietary intake. This pre-registered study examined the impact of online exposure to images of natural scenes on delay discounting and food preferences. It was predicted that exposure to images of natural scenes (vs. images of urban scenes) would be associated with: (i) lower delay discounting; (ii) higher desirability for fruits and vegetables (and lower desirability for more energy-dense foods); and (iii) delay discounting would mediate the effect of nature-image exposure on food desirability. Adult participants (N = 109) were recruited to an online between-subjects experiment in which they viewed a timed sequence of six images either showing natural landscape scenes or urban scenes. They then completed measures of mood, delay discounting (using a five-trial hypothetical monetary discounting task) and rated their momentary desire to eat four fruits and vegetables (F&V), and four energy-dense foods. There was no statistically significant effect of experimental condition (natural vs. urban image exposure) on delay discounting or food desirability. Bayes factors supported the null hypothesis for discounting (BF01 = 4.89), and energy-dense food desirability (BF01 = 7.21), but provided no strong evidence for either hypothesis for F&V desirability (BF01 = 0.78). These findings indicate that brief online exposure to images of nature does not affect momentary impulsivity or energy-dense food preference, whereas for preference for less-energy dense foods, the evidence was inconclusive.

Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images

Semantic segmentation from very fine resolution (VFR) urban scene images plays a significant role in several application scenarios including autonomous driving, land cover classification, urban planning, etc. However, the tremendous details contained in the VFR image, especially the considerable variations in scale and appearance of objects, severely limit the potential of the existing deep learning approaches. Addressing such issues represents a promising research field in the remote sensing community, which paves the way for scene-level landscape pattern analysis and decision making. In this paper, we propose a Bilateral Awareness Network which contains a dependency path and a texture path to fully capture the long-range relationships and fine-grained details in VFR images. Specifically, the dependency path is conducted based on the ResT, a novel Transformer backbone with memory-efficient multi-head self-attention, while the texture path is built on the stacked convolution operation. In addition, using the linear attention mechanism, a feature aggregation module is designed to effectively fuse the dependency features and texture features. Extensive experiments conducted on the three large-scale urban scene image segmentation datasets, i.e., ISPRS Vaihingen dataset, ISPRS Potsdam dataset, and UAVid dataset, demonstrate the effectiveness of our BANet. Specifically, a 64.6% mIoU is achieved on the UAVid dataset.

Lattice-Point Mutually Guided Ground-to-Aerial Feature Matching for Urban Scene Images

Ground-to-aerial feature matching bridges information from cross-view images, which enables optimized urban applications, e.g., pixel-level geolocating and complete urban 3-D reconstruction. However, urban ground and aerial images typically suffer from drastic changes in viewpoint, scale, and illumination, together with repetitive patterns. Thus, direct matching of local features between ground and aerial images is particularly difficult because of the low similarity of local descriptors and high ambiguity in true–false match discrimination. For this challenging task, we propose a novel lattice-point mutually guided matching (LPMG) method in this article. We specifically address two key issues: 1) reducing descriptor variance and 2) enhancing true–false match discriminability. The former is solved by recovering the geometry and appearance of the underlying image region in 3-D through automatic view rectification on ground and aerial images. The latter is circumvented by replacing the conventional mismatch removal with an LPMG strategy. In this strategy, the topology structure of repeated façade elements (i.e., lattice), and the high reliable point matching seeds, are first extracted from the rectified ground and aerial images. Then, the point matching seeds guide the self-similar lattice tiles from two views to be precisely aligned, thereby estimating an accurate transformation model from lattice tile correspondences. Finally, the estimated model powerfully supervises the differentiation of true and false matches from the entire putative match set. Extensive experiments conducted on several datasets show that our method can obtain a considerable number of nearly pure correct matches from urban ground and aerial images, significantly outperforming those existing methods.

Parsing very high resolution urban scene images by learning deep ConvNets with edge-aware loss

Parsing very high resolution (VHR) urban scene images into regions with semantic meaning, e.g. buildings and cars, is a fundamental task in urban scene understanding. However, due to the huge quantity of details contained in an image and the large variations of objects in scale and appearance, the existing semantic segmentation methods often break one object into pieces, or confuse adjacent objects and thus fail to depict these objects consistently. To address these issues uniformly, we propose a standalone end-to-end edge-aware neural network (EaNet) for urban scene semantic segmentation. For semantic consistency preservation inside objects, the EaNet model incorporates a large kernel pyramid pooling (LKPP) module to capture rich multi-scale context with strong continuous feature relations. To effectively separate confusing objects with sharp contours, a Dice-based edge-aware loss function (EA loss) is devised to guide the EaNet to refine both the pixel- and image-level edge information directly from semantic segmentation prediction. In the proposed EaNet model, the LKPP and the EA loss couple to enable comprehensive feature learning across an entire semantic object. Extensive experiments on three challenging datasets demonstrate that our method can be readily generalized to multi-scale ground/aerial urban scene images, achieving 81.7% in mIoU on Cityscapes Test set and 90.8% in the mean F1-score on the ISPRS Vaihingen 2D Test set. Code is available at: https://github.com/geovsion/EaNet.

Adversarial Attack Against Urban Scene Segmentation for Autonomous Vehicles

Understanding the surrounding environment is crucial for autonomous vehicles to make correct driving decisions. In particular, urban scene segmentation is a significant integral module commonly equipped in the perception system of autonomous vehicles to understand the real scene like a human. Any missegmentation of the driving scenario can potentially result in uncontrollable consequences such as serious accidents or the exception of the perception system. In this article, we investigate the vulnerability of the popular scene segmentation models designed with the backbones of deep neural networks (DNNs), which have been shown to be sensitive to adversarial attacks. Specifically, we propose an iterative projected gradient-based attack method that can effectively fool several DNN-based segmentation models with a remarkably higher attacking successful rate, and much smaller adversarial perturbations. Moreover, we also develop an adversarial training algorithm with min-max optimization style to enrich the robustness of the scene segmentation models. Extensive experiments on the Cityscape benchmark dataset consisting of large-scale urban scene images for autonomous vehicles demonstrate the effectiveness of our proposed attack method, as well as the benefit of the adversarial training scheme for the scene segmentation models.

GF-2 Panchromatic and Multispectral Remote Sensing Image Registration Algorithm

Image registration is an important step in remote sensing image preprocessing. The accuracy, efficiency, and automatic degree of image registration will directly affect the results of subsequent image processing and analysis. Due to the slow registration speed and low registration accuracy of SIFT algorithm for GF-2 panchromatic and multispectral remote sensing images, SIFT algorithm is improved in this article to improve the efficiency of image registration algorithm. In this article, the strategy of information entropy meshes is introduced, and feature extraction is carried out only for the regions with large information entropy, so that the running time of feature extraction can be reduced. By introducing Canny operator to eliminate the unstable edge response points, the number of feature points can be further reduced, thus the computation amount of the algorithm can be also reduced. Due to the poor registration effect of SIFT algorithm for remote sensing images, a new gradient calculation method and feature description method are used in this article to enhance the robustness of feature descriptors. In the feature matching stage, this article proposes a two-layer matching strategy, which uses mutual Euclidean distance for initial matching, then uses FSC algorithm for fine matching, and finally realizes the registration of GF-2 panchromatic and multispectral remote sensing images. Experimental results show that the proposed algorithm can obtain more correct matching point pairs when the number of extracted feature points is small. Moreover, the registration accuracy and speed are both higher than the comparison algorithm. This article uses the remote sensing images of urban scenes dominated by plains and mountainous scenes with small terrain fluctuation respectively for experiments. The algorithm in this article can achieve registration accuracy better than 0.5 pixels, and in terms of time consumption, it is only about 70% of SIFT algorithm, and the computational efficiency is better than SIFT algorithm, which can meet the actual GF-2 Pan and MSI registration tasks.

Urban scene semantic segmentation with insufficient labeled data

Semantic segmentation of urban scenes is an enabling factor for a wide range of applications. With the development of deep learning in recent years, semantic segmentation tasks using high-capacity models have achieved considerable successes on large datasets. However, the pixel-level annotation process, especially for urban scene images with various objects, is tedious and labor intensive. Meanwhile, the scale of the unlabeled data, which is currently easy to collect, is often much larger than labeled data. Thus, using the abundant unlabeled data to make up the loss of the segmentation model from insufficient labeled data is of great interest. In this paper, we propose a semi-supervised method based on reinforcement learning to capture the contextual information from the unlabeled data to improve the model trained on the small scale labeled data. Both quantitative and qualitative experiments have shown the effectiveness of the proposed method.

URBAN-i: From urban scenes to mapping slums, transport modes, and pedestrians in cities using deep learning and computer vision

In recent years, deep learning and computer vision have been applied to solve complex problems across many domains. In urban studies, these technologies have been instrumental in the development of smart cities and autonomous vehicles. However, a knowledge gap is present when it comes to informal urban regions in less developed countries. How can deep learning and artificial intelligence untangle the complexities of informality to advance urban modelling? In this paper, we introduce a framework for multipurpose realistic-dynamic urban modelling using deep convolutional neural networks. The purpose of the framework is twofold: (1) to sense and detect informality and slums in urban scenes from aerial and street-level images and (2) to detect pedestrian and transport modes. The model has been trained on images of urban scenes in cities across the globe. The framework shows strong validation performance in the identification of planned and unplanned regions, despite broad variations in the classified images. The algorithms of the URBAN-i model are coded in Python and the trained models can be applied to images of any urban setting, including informal settlements and slum regions.

A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes.

During the last half decade, convolutional neural networks (CNNs) have triumphed over semantic segmentation, which is one of the core tasks in many applications such as autonomous driving and augmented reality. However, to train CNNs requires a considerable amount of data, which is difficult to collect and laborious to annotate. Recent advances in computer graphics make it possible to train CNNs on photo-realistic synthetic imagery with computer-generated annotations. Despite this, the domain mismatch between real images and the synthetic data hinders the models' performance. Hence, we propose a curriculum-style learning approach to minimizing the domain gap in urban scene semantic segmentation. The curriculum domain adaptation solves easy tasks first to infer necessary properties about the target domain; in particular, the first task is to learn global label distributions over images and local distributions over landmark superpixels. These are easy to estimate because images of urban scenes have strong idiosyncrasies (e.g., the size and spatial relations of buildings, streets, cars, etc.). We then train a segmentation network, while regularizing its predictions in the target domain to follow those inferred properties. In experiments, our method outperforms the baselines on two datasets and three backbone networks. We also report extensive ablation studies about our approach.

Gaze-based signatures of mind wandering during real-world scene processing.

Physiological limitations on the visual system require gaze to move from location to location to extract the most relevant information within a scene. Therefore, gaze provides a real-time index of the information-processing priorities of the visual system. We investigated gaze allocation during mind wandering (MW), a state where cognitive priorities shift from processing task-relevant external stimuli (i.e., the visual world) to task-irrelevant internal thoughts. In both a main study and a replication, we recorded the eye movements of college-aged adults who studied images of urban scenes and responded to pseudorandom thought probes on whether they were mind wandering or attentively viewing at the time of the probe. Probe-caught MW was associated with fewer and longer fixations, greater fixation dispersion, and more frequent eyeblinks (only observed in the main study) relative to periods of attentive scene viewing. These findings demonstrate that gaze indices typically considered to represent greater engagement with scene processing (e.g., longer fixations) can also indicate MW. In this way, the current work exhibits a need for empirical investigations and computational models of gaze control to account for MW for a more accurate representation of the moment-to-moment information-processing priorities of the visual system. (PsycINFO Database Record

INVESTIGATION OF JOINT VISIBILITY BETWEEN SAR AND OPTICAL IMAGES OF URBAN ENVIRONMENTS

Abstract. In this paper, we present a work-flow to investigate the joint visibility between very-high-resolution SAR and optical images of urban scenes. For this task, we extend the simulation framework SimGeoI to enable a simulation of individual pixels rather than complete images. Using the extended SimGeoI simulator, we carry out a case study using a TerraSAR-X staring spotlight image and a Worldview-2 panchromatic image acquired over the city of Munich, Germany. The results of this study indicate that about 55 % of the scene are visible in both images and are thus suitable for matching and data fusion endeavours, while about 25 % of the scene are affected by either radar shadow or optical occlusion. Taking the image acquisition parameters into account, our findings can provide support regarding the definition of upper bounds for image fusion tasks, as well as help to improve acquisition planning with respect to different application goals.

The effect of assuming static or dynamic gaze behaviour on the estimated background luminance of drivers

Application of the Commission Internationale de l’Eclairage system for mesopic photometry requires that an estimate is made of the observers’ state of luminance adaptation. This paper addresses an assumption made when estimating background luminance, a component of adaptation luminance. Specifically, using spatial sampling of the visual field we compare background luminances determined from assumptions of static or dynamic visual gaze, the former being a simplification, the latter being a better representation based on eye movements when driving. The comparison was undertaken with luminance images of urban scenes at night on three roads, two real and one simulated. It was found that background luminances were significantly higher when estimated using the dynamic assumption. It was also found that scene luminances at the point of foveal fixations tend to be higher than those luminances influencing peripheral regions of the retina. Compared with the background luminance estimated for a dynamic peripheral field, a horizon-centred 10° circle led to a slightly higher estimate and the road surface luminance to a slightly lower estimate of background luminance.