Semantic Segmentation Of Aerial Images Research Articles

Towards sustainable coastal management: aerial imagery and deep learning for high-resolution Sargassum mapping

The massive arrival of pelagic Sargassum on the coasts of several countries of the Atlantic Ocean began in 2011 and to date continues to generate social and environmental challenges for the region. Therefore, knowing the distribution and quantity of Sargassum in the ocean, coasts, and beaches is necessary to understand the phenomenon and develop protocols for its management, use, and final disposal. In this context, the present study proposes a methodology to calculate the area Sargassum occupies on beaches in square meters, based on the semantic segmentation of aerial images using the pix2pix architecture. For training and testing the algorithm, a unique dataset was built from scratch, consisting of 15,268 aerial images segmented into three classes. The images correspond to beaches in the cities of Mahahual and Puerto Morelos, located in Quintana Roo, Mexico. To analyze the results the fβ-score metric was used. The results for the Sargassum class indicate that there is a balance between false positives and false negatives, with a slight bias towards false negatives, which means that the algorithm tends to underestimate the Sargassum pixels in the images. To know the confidence intervals within which the algorithm performs better, the results of the f0.5-score metric were resampled by bootstrapping considering all classes and considering only the Sargassum class. From the above, we found that the algorithm offers better performance when segmenting Sargassum images on the sand. From the results, maps showing the Sargassum coverage area along the beach were designed to complement the previous ones and provide insight into the field of study.

3D Modeling of rural environments from multiscale aerial imagery

Given the increasing attention to environmental preservation and sustainable development, the digitization of rural landscapes stands out as a pivotal strategy for effective environmental management and sustainability, land use planning, and preservation of cultural heritage. This work proposes a novel methodology for generating 3D models of rural landscapes by integrating multiscale data sources. Although Unmanned Aerial Vehicles (UAV) simplify the acquisition of multi-source data, their coverage is typically restricted to small landscapes due to their limited range and flight time. On the other hand, although the use of aerial images provides a broader view of the terrain, it is important to note that the low resolution of these images interferes with the task of accurate 3D modeling. Given these challenges, we propose a methodology that combines UAV data and high-resolution aerial imagery provided by the Spanish National Orthophoto Program (PNOA). This multi-source data integration is crucial to generating detailed and accurate 3D models of rural environments. The proposed methodology involves three steps: (1) semantic segmentation of aerial images identifying features such as vegetation, ground, and human-made structures, (2) estimation of the Digital Elevation Model (DEM), and (3) 3D modeling of rural environments using the point clouds generated from UAV images. The conducted experiments demonstrate the effectiveness of our approach identifying and representing previously mentioned features. Thus, this work presents advances in 3D representation techniques for real scenarios, contributing to the coordination of land utilization and environmental sustainability in rural landscapes.

Aerial image semantic segmentation based on 3D fits a small dataset of 1D

<p>Time restrictions and lack of precision demand that the initial technique be abandoned. Even though the remaining datasets had fewer identified classes than initially planned for the study, the labels were more accurate. Because of the need for additional data, a single network cannot categorize all the essential elements in a picture, including bodies of water, roads, trees, buildings, and crops. However, the final network gains some invariance in detecting these classes with environmental changes due to the different geographic positions of roads and buildings discovered in the final datasets, which could be valuable in future navigation research. At the moment, binary classifications of a single class are the only datasets that can be used for the semantic segmentation of aerial images. Even though some pictures have more than one classification, images of roads and buildings were only found in a significant number of samples. Then, the building datasets were pooled to produce a larger dataset and for the constructed models to gain some invariance on image location. Because of the massive disparity in sample size, road datasets needed to be integrated.</p>

An improved beluga whale optimizer-Derived Adaptive multi-channel DeepLabv3+ for semantic segmentation of aerial images.

Semantic segmentation process over Remote Sensing images has been regarded as hot research work. Even though the Remote Sensing images provide many essential features, the sampled images are inconsistent in size. Even if a similar network can segment Remote Sensing images to some extents, segmentation accuracy needs to be improved. General neural networks are used to improve categorization accuracy, but they also caused significant losses to target scale and spatial features, and the traditional common features fusion techniques can only resolve some of the issues. A segmentation network has been designed to resolve the above-mentioned issues as well. With the motive of addressing the difficulties in the existing semantic segmentation techniques for aerial images, the adoption of deep learning techniques is utilized. This model has adopted a new Adaptive Multichannel Deeplabv3+ (AMC-Deeplabv3+) with the help of a new meta-heuristic algorithm called Improved Beluga whale optimization (IBWO). Here, the hyperparameters of Multichannel deeplabv3+ are optimized by the IBWO algorithm. The proposed model significantly enhances the performance of the overall system by measuring the accuracy and dice coefficient. The proposed model attains improved accuracies of 98.65% & 98.72% for dataset 1 and 2 respectively and also achieves the dice coefficient of 98.73% & 98.85% respectively with a computation time of 113.0123 seconds. The evolutional outcomes of the proposed model show significantly better than the state of the art techniques like CNN, MUnet and DFCNN models.

An adaptive multichannel DeepLabv3 + for semantic segmentation of aerial images using improved Beluga Whale Optimization Algorithm

An adaptive multichannel DeepLabv3 + for semantic segmentation of aerial images using improved Beluga Whale Optimization Algorithm

MSINet: Mining scale information from digital surface models for semantic segmentation of aerial images

Compared with other kinds of images, aerial images have more obvious object scale distinction and larger resolution, which results in that the whole scale information of aerial images can hardly be explored. To address this difficulty, we develop a novel network based on the digital surface models (DSMs) of aerial images in this paper. The proposed network termed MSINet can efficiently mine scale information through the DSMs from two aspects. Firstly, we propose an interpolation pyramid algorithm to encode the scale information from the DSMs and hence provide a scale prior information to the normal segmentation network. The interpolation pyramid algorithm implements interpolation operations with different scales on the DSMs and detects the pixel value change after the interpolation operations. Objects with different scales will express diverse changes, which provides useful information to encode their scale information. Secondly, aiming to address the problem that the DSMs contain a large amount of noise in the boundary part, a spatial information enhancement module and a mutual-guidance module are developed in this paper. These two modules can fix the misleading guidance information caused by the noise in the boundary part of the DSMs and hence achieve more accurate scale information inserting. The extensive experimental results prove that our methods can outperform other competitors in terms of qualitative and quantitative performance.

A Data-Related Patch Proposal for Semantic Segmentation of Aerial Images

A Data-Related Patch Proposal for Semantic Segmentation of Aerial Images

An Adaptive DeepLabv3+ for Semantic Segmentation of Aerial Images Using Improved Golden Eagle Optimization Algorithm

An Adaptive DeepLabv3+ for Semantic Segmentation of Aerial Images Using Improved Golden Eagle Optimization Algorithm

Boosting Semantic Segmentation of Aerial Images via Decoupled and Multilevel Compaction and Dispersion

Boosting Semantic Segmentation of Aerial Images via Decoupled and Multilevel Compaction and Dispersion

Cycle and Self-Supervised Consistency Training for Adapting Semantic Segmentation of Aerial Images

Semantic segmentation is a critical problem for many remote sensing (RS) image applications. Benefiting from large-scale pixel-level labeled data and the continuous evolution of deep neural network architectures, the performance of semantic segmentation approaches has been constantly improved. However, deploying a well-trained model on unseen and diverse testing environments remains a major challenge: a large gap between data distributions in train and test domains results in severe performance loss, while manual dense labeling is costly and not scalable. To this end, we proposed an unsupervised domain adaptation framework for RS image semantic segmentation that is both practical and effective. The framework is supported by the consistency principle, including the cycle consistency in the input space and self-supervised consistency in the training stage. Specifically, we introduce cycle-consistent generative adversarial networks to reduce the discrepancy between source and target distributions by translating one into the other. The translated source data then drive a pipeline of supervised semantic segmentation model training. We enforce consistency of model predictions across target image transformations in order to provide self-supervision for the unlabeled target data. Experiments and extensive ablation studies demonstrate the effectiveness of the proposed approach on two challenging benchmarks, on which we achieve up to 9.95% and 7.53% improvements in accuracy over the state-of-the-art methods, respectively.

Research Contribution and Comprehensive Review towards the Semantic Segmentation of Aerial Images Using Deep Learning Techniques

Semantic segmentation is a significant research topic for decades and has been employed in several applications. In recent years, semantic segmentation has been focused on different deep learning approaches in the area of computer vision, which has aimed for getting superior efficiency while analyzing the aerial and remote-sensing images. The main aim of this review is to provide a clear algorithmic categorization and analysis of the diverse contribution of semantic segmentation of aerial images and expects to give the comprehensive details associated with the recent developments. In addition, the emerged deep learning methods demonstrated much improved performance measures on several public datasets and incredible efforts have been dedicated to advancing pixel-level accuracy. Hence, the analysis on diverse datasets of each contribution is studied, and also, the best performance measures achieved by the existing semantic segmentation models are evaluated. Thus, this survey can facilitate researchers in understanding the development of semantic segmentation in a shorter time, simplify understanding of its latest advancements, research gaps, and challenges to be used as a reference for developing the new semantic image segmentation models in the future.

MLCRNet: Multi-Level Context Refinement for Semantic Segmentation in Aerial Images

In this paper, we focus on the problem of contextual aggregation in the semantic segmentation of aerial images. Current contextual aggregation methods only aggregate contextual information within specific regions to improve feature representation, which may yield poorly robust contextual information. To address this problem, we propose a novel multi-level context refinement network (MLCRNet) that aggregates three levels of contextual information effectively and efficiently in an adaptive manner. First, we designed a local-level context aggregation module to capture local information around each pixel. Second, we integrate multiple levels of context, namely, local-level, image-level, and semantic-level, to aggregate contextual information from a comprehensive perspective dynamically. Third, we propose an efficient multi-level context transform (EMCT) module to address feature redundancy and to improve the efficiency of our multi-level contexts. Finally, based on the EMCT module and feature pyramid network (FPN) framework, we propose a multi-level context feature refinement (MLCR) module to enhance feature representation by leveraging multi-level contextual information. Extensive empirical evidence demonstrates that our MLCRNet achieves state-of-the-art performance on the ISPRS Potsdam and Vaihingen datasets.

Semantic Segmentation of Aerial Images Using U-Net Architecture

Arial images are very high resolution. The automation for map generation and semantic segmentation of aerial images are challenging problems in semantic segmentation. The semantic segmentation process does not give us precise details of the remote sensing images due to the low resolution of the aerial images. Hence, we propose an algorithm U-Net Architecture to solve this problem. It is classified into two paths. The compression path (also called: the encoder) is the first path and is used to capture the image’s context. The encoder is just a convolutional and maximal pooling layer stack. The symmetric expanding path (also called: the decoder) is the second path, which is used to enable exact localization by transposed convolutions. This task is commonly referred to as dense prediction, which is completely connected to each other and also with the former neurons which gives rise to dense layers. Thus it is an end-to-end fully convolutional network (FCN), i.e. it only contains convolutional layers and does not contain any dense layer because of which it can accept images of any size. The performance of the model will be evaluated by improving the image using the proposed method U-NET and obtaining an improved image by measuring the accuracy compared with the value of accuracy with previous methods.

Geometry-Aware Segmentation of Remote Sensing Images via Joint Height Estimation

Recent studies have shown the benefits of using additional elevation data [e.g., digital surface model (DSM) or normalized DSM (nDSM)] for enhancing the performance of the semantic labeling of aerial images. However, previous methods mostly adopt 3-D elevation information as additional inputs, while, in many real-world applications, one does not have the corresponding DSM images at hand, and the spatial resolution of acquired DSM images usually does not match the aerial images. To alleviate this data constraint and also take advantage of 3-D elevation information, in this letter, a geometry-aware segmentation model is introduced to achieve accurate semantic labeling of aerial images via joint height estimation. Instead of using a single-stream encoder-decoder network for semantic labeling, we design a separate decoder branch to predict the height map and use the DSM images as side supervision to train this newly designed decoder branch. With the newly designed decoder branch, our model can distill the 3-D geometric features from 2-D appearance features under the supervision of ground-truth DSM images. Moreover, we develop a new geometry-aware convolution module that fuses the 3-D geometric features from the height decoder branch and the 2-D contextual features from the semantic segmentation branch. The fused feature embeddings can produce geometry-aware segmentation maps with enhanced performance. Our model is trained with DSM images as side supervision, while, in the inference stage, it does not require DSM data and directly predicts the semantic labels. Experiments on International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam data sets demonstrate the effectiveness of the proposed method for the semantic segmentation of aerial images.

Unsupervised Semantic Segmentation of Aerial Images With Application to UAV Localization

Recent advances and applications of aerial image semantic segmentation have yielded an increase in their use in day-to-day tasks. However, state-of-art algorithms, composed mostly of deep semantic segmentation networks (DSSNs), may not be suitable for application domains in which labels (targeted output masks) are scarce. This work proposes a fully unsupervised semantic segmentation method able to find the appropriate number of semantics labels (yielded from clustering) without the need of a previously annotated dataset. Once the semantic labels are identified, a classifier trained from these labels can be used to assign semantics to new input images. In contrast to other clustering-based segmentation methods, our method does not require any input parameter to find the clustering partition that best accommodates data. Empirical results on two real datasets showed that the use of semantics in a template-matching-based unmanned aerial vehicle (UAV) localization pipeline reduced the overall position estimation error.

Class-Incremental Semantic Segmentation of Aerial Images via Pixel-Level Feature Generation and Task-Wise Distillation

Deep neural networks achieve significant progress in semantic segmentation but still suffer from the catastrophic forgetting problem, i.e., networks will forget old classes as they learn new ones. In this article, we propose an effective class-incremental segmentation framework without storing old data. Specifically, to alleviate the issue of catastrophic forgetting, we present two important modules, i.e., the Pixel-level Feature Generation (PFG) module, and the Task-wise Knowledge distillation (TKD) module. The PFG module is designed to constantly generate any number of features of the old classes to keep the old memory. The PFG module is the first attempt to use the generative method in class-incremental segmentation of aerial images, and it abandons the previous image generation approach but to generate pixel-level features, which is more suitable for the segmentation task. Meanwhile, the proposed TKD module is specially designed for class incremental tasks, and it only compares classes in the same learning step (task), thus avoiding the squeezing of new classes to old classes when the output is normalized (softmax), making distillation more effective. Sufficient experiments show that our method is remarkably effective and achieves more than 4.5% gains compared with state-of-the-art methods, and more than 13% compared with baselines, on all learning conditions. The ablation studies show that the PFG module and the TKD module are both indispensables. Besides, the proposed framework can be well combined with any existing class incremental learning method to achieve better performance.

Semantic Segmentation of Remote Sensing Images With Sparse Annotations

Training Convolutional Neural Networks (CNNs) for very high resolution images requires a large quantity of high-quality pixel-level annotations, which is extremely labor- and time-consuming to produce. Moreover, professional photo interpreters might have to be involved for guaranteeing the correctness of annotations. To alleviate such a burden, we propose a framework for semantic segmentation of aerial images based on incomplete annotations, where annotators are asked to label a few pixels with easy-to-draw scribbles. To exploit these sparse scribbled annotations, we propose the FEature and Spatial relaTional regulArization (FESTA) method to complement the supervised task with an unsupervised learning signal that accounts for neighbourhood structures both in spatial and feature terms.

SEMANTIC SEGMENTATION OF AIRBORNE IMAGES AND CORRESPONDING DIGITAL SURFACE MODELS – ADDITIONAL INPUT DATA OR ADDITIONAL TASK?

Abstract. We analyze the effects of additional height data for semantic segmentation of aerial images with a convolutional encoder-decoder network. Besides a merely image-based semantic segmentation, we trained the same network with height as additional input and furthermore, we defined a multi-task model, where we trained the network to estimate the relative height of objects in parallel to semantic segmentation on the image data only. Our findings are, that excellent results are possible for image data only and additional height information has no significant effect – neither when employed as extra input nor when used for multi-task training, even with differently weighted losses. Based on our results, we, thus, hypothesize that a strong encoder-decoder network implicitly learns the correlation of object categories and relative heights.

Unsupervised Domain Adaptation Using Generative Adversarial Networks for Semantic Segmentation of Aerial Images

Segmenting aerial images is of great potential in surveillance and scene understanding of urban areas. It provides a mean for automatic reporting of the different events that happen in inhabited areas. This remarkably promotes public safety and traffic management applications. After the wide adoption of convolutional neural networks methods, the accuracy of semantic segmentation algorithms could easily surpass 80% if a robust dataset is provided. Despite this success, the deployment of a pretrained segmentation model to survey a new city that is not included in the training set significantly decreases accuracy. This is due to the domain shift between the source dataset on which the model is trained and the new target domain of the new city images. In this paper, we address this issue and consider the challenge of domain adaptation in semantic segmentation of aerial images. We designed an algorithm that reduces the domain shift impact using generative adversarial networks (GANs). In the experiments, we tested the proposed methodology on the International Society for Photogrammetry and Remote Sensing (ISPRS) semantic segmentation dataset and found that our method improves overall accuracy from 35% to 52% when passing from the Potsdam domain (considered as source domain) to the Vaihingen domain (considered as target domain). In addition, the method allows efficiently recovering the inverted classes due to sensor variation. In particular, it improves the average segmentation accuracy of the inverted classes due to sensor variation from 14% to 61%.

Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images

When approaching the semantic segmentation of overhead imagery in the decimeter spatial resolution range, successful strategies usually combine powerful methods to learn the visual appearance of the semantic classes (e.g. convolutional neural networks) with strategies for spatial regularization (e.g. graphical models such as conditional random fields).In this paper, we propose a method to learn evidence in the form of semantic class likelihoods, semantic boundaries across classes and shallow-to-deep visual features, each one modeled by a multi-task convolutional neural network architecture. We combine this bottom-up information with top-down spatial regularization encoded by a conditional random field model optimizing the label space across a hierarchy of segments with constraints related to structural, spatial and data-dependent pairwise relationships between regions.Our results show that such strategy provide better regularization than a series of strong baselines reflecting state-of-the-art technologies. The proposed strategy offers a flexible and principled framework to include several sources of visual and structural information, while allowing for different degrees of spatial regularization accounting for priors about the expected output structures.