Semantic Segmentation Of Remote Sensing Images Research Articles

Active Bidirectional Self-Training Network for Cross-Domain Segmentation in Remote-Sensing Images

Semantic segmentation with cross-domain adaptation in remote-sensing images (RSIs) is crucial and mitigates the expense of manually labeling target data. However, the performance of existing unsupervised domain adaptation (UDA) methods is still significantly impacted by domain bias, leading to a considerable gap compared to supervised trained models. To address this, our work focuses on semi-supervised domain adaptation, selecting a small subset of target annotations through active learning (AL) that maximize information to improve domain adaptation. Overall, we propose a novel active bidirectional self-training network (ABSNet) for cross-domain semantic segmentation in RSIs. ABSNet consists of two sub-stages: a multi-prototype active region selection (MARS) stage and a source-weighted class-balanced self-training (SCBS) stage. The MARS approach captures the diversity in labeled source data by introducing multi-prototype density estimation based on Gaussian mixture models. We then measure inter-domain similarity to select complementary and representative target samples. Through fine-tuning with the selected active samples, we propose an enhanced self-training strategy SCBS, designed for weighted training on source data, aiming to avoid the negative effects of interfering samples. We conduct extensive experiments on the LoveDA and ISPRS datasets to validate the superiority of our method over existing state-of-the-art domain-adaptive semantic segmentation methods.

ST-MDAMNet: Swin transformer combines multi-dimensional attention mechanism for semantic segmentation of high-resolution earth surface images

In the derection of remote sensing (RS) image analysis, semantic segmentation, as an important technology, is of key significance for the identification and analysis of land surface cover types. In recent years, applying deep learning models to tasks such as road extraction, water distribution extraction, building classification and building segmentation from RS images has become an important research hotspot. Due to its limited receptive field, traditional convolutional neural networks (CNN) cannot effectively capture global context information. Transformer uses the multi-head self-attention mechanism to capture a wide range of information and can solve this problem well. Therefore, we proposed ST-MDAMNet based on Swin Transformer and combined with the multi-dimensional attention mechanism. First, a feature enhancement module (FAM) is introduced after each stage of the Swin Transformer encoder to effectively enhance the model’s proficiency in identifying essential information. Secondly, a feature fusion module (FFM) is proposed to effectively fuse the multi-scale information of the encoder part. It further improves the expression ability of different dimensional features and effectively improves the detection effect of small targets. Ultimately, the fused features are input into the multi-dimensional attention module (MDAM) to carefully optimize the features, which greatly increases the effect of semantic segmentation of RS images. We demonstrate the effectiveness of each module through ablation experiments. Comparative experiments are completed on two publicly large-scale datasets, and the proposed method shows excellent results compared with state-of-the-art methods.

Mask2Former with Improved Query for Semantic Segmentation in Remote-Sensing Images

Semantic segmentation of remote sensing (RS) images is vital in various practical applications, including urban construction planning, natural disaster monitoring, and land resources investigation. However, RS images are captured by airplanes or satellites at high altitudes and long distances, resulting in ground objects of the same category being scattered in various corners of the image. Moreover, objects of different sizes appear simultaneously in RS images. For example, some objects occupy a large area in urban scenes, while others only have small regions. Technically, the above two universal situations pose significant challenges to the segmentation with a high quality for RS images. Based on these observations, this paper proposes a Mask2Former with an improved query (IQ2Former) for this task. The fundamental motivation behind the IQ2Former is to enhance the capability of the query of Mask2Former by exploiting the characteristics of RS images well. First, we propose the Query Scenario Module (QSM), which aims to learn and group the queries from feature maps, allowing the selection of distinct scenarios such as the urban and rural areas, building clusters, and parking lots. Second, we design the query position module (QPM), which is developed to assign the image position information to each query without increasing the number of parameters, thereby enhancing the model’s sensitivity to small targets in complex scenarios. Finally, we propose the query attention module (QAM), which is constructed to leverage the characteristics of query attention to extract valuable features from the preceding queries. Being positioned between the duplicated transformer decoder layers, QAM ensures the comprehensive utilization of the supervisory information and the exploitation of those fine-grained details. Architecturally, the QSM, QPM, and QAM as well as an end-to-end model are assembled to achieve high-quality semantic segmentation. In comparison to the classical or state-of-the-art models (FCN, PSPNet, DeepLabV3+, OCRNet, UPerNet, MaskFormer, Mask2Former), IQ2Former has demonstrated exceptional performance across three publicly challenging remote-sensing image datasets, 83.59 mIoU on the Vaihingen dataset, 87.89 mIoU on Potsdam dataset, and 56.31 mIoU on LoveDA dataset. Additionally, overall accuracy, ablation experiment, and visualization segmentation results all indicate IQ2Former validity.

Intra- and Inter-Image Causal Intervention for Robust Semantic Segmentation in Remote-Sensing Images

Intra- and Inter-Image Causal Intervention for Robust Semantic Segmentation in Remote-Sensing Images

SSCNet: A Spectrum-Space Collaborative Network for Semantic Segmentation of Remote Sensing Images

Semantic segmentation plays a pivotal role in the intelligent interpretation of remote sensing images (RSIs). However, conventional methods predominantly focus on learning representations within the spatial domain, often resulting in suboptimal discriminative capabilities. Given the intrinsic spectral characteristics of RSIs, it becomes imperative to enhance the discriminative potential of these representations by integrating spectral context alongside spatial information. In this paper, we introduce the spectrum-space collaborative network (SSCNet), which is designed to capture both spectral and spatial dependencies, thereby elevating the quality of semantic segmentation in RSIs. Our innovative approach features a joint spectral–spatial attention module (JSSA) that concurrently employs spectral attention (SpeA) and spatial attention (SpaA). Instead of feature-level aggregation, we propose the fusion of attention maps to gather spectral and spatial contexts from their respective branches. Within SpeA, we calculate the position-wise spectral similarity using the complex spectral Euclidean distance (CSED) of the real and imaginary components of projected feature maps in the frequency domain. To comprehensively calculate both spectral and spatial losses, we introduce edge loss, Dice loss, and cross-entropy loss, subsequently merging them with appropriate weighting. Extensive experiments on the ISPRS Potsdam and LoveDA datasets underscore SSCNet’s superior performance compared with several state-of-the-art methods. Furthermore, an ablation study confirms the efficacy of SpeA.

Deep Learning Methods for Semantic Segmentation in Remote Sensing with Small Data: A Survey

The annotations used during the training process are crucial for the inference results of remote sensing images (RSIs) based on a deep learning framework. Unlabeled RSIs can be obtained relatively easily. However, pixel-level annotation is a process that necessitates a high level of expertise and experience. Consequently, the use of small sample training methods has attracted widespread attention as they help alleviate reliance on large amounts of high-quality labeled data and current deep learning methods. Moreover, research on small sample learning is still in its infancy owing to the unique challenges faced when completing semantic segmentation tasks with RSI. To better understand and stimulate future research that utilizes semantic segmentation tasks with small data, we summarized the supervised learning methods and challenges they face. We also reviewed the supervised approaches with data that are currently popular to help elucidate how to efficiently utilize a limited number of samples to address issues with semantic segmentation in RSI. The main methods discussed are self-supervised learning, semi-supervised learning, weakly supervised learning and few-shot methods. The solution of cross-domain challenges has also been discussed. Furthermore, multi-modal methods, prior knowledge constrained methods, and future research required to help optimize deep learning models for various downstream tasks in relation to RSI have been identified.

Dynamic High-Resolution Network for Semantic Segmentation in Remote-Sensing Images

Semantic segmentation of remote-sensing (RS) images is one of the most fundamental tasks in the understanding of a remote-sensing scene. However, high-resolution RS images contain plentiful detailed information about ground objects, which scatter everywhere spatially and have variable sizes, styles, and visual appearances. Due to the high similarity between classes and diversity within classes, it is challenging to obtain satisfactory and accurate semantic segmentation results. This paper proposes a Dynamic High-Resolution Network (DyHRNet) to solve this problem. Our proposed network takes HRNet as a super-architecture, aiming to leverage the important connections and channels by further investigating the parallel streams at different resolution representations of the original HRNet. The learning task is conducted under the framework of a neural architecture search (NAS) and channel-wise attention module. Specifically, the Accelerated Proximal Gradient (APG) algorithm is introduced to iteratively solve the sparse regularization subproblem from the perspective of neural architecture search. In this way, valuable connections are selected for cross-resolution feature fusion. In addition, a channel-wise attention module is designed to weight the channel contributions for feature aggregation. Finally, DyHRNet fully realizes the dynamic advantages of data adaptability by combining the APG algorithm and channel-wise attention module simultaneously. Compared with nine classical or state-of-the-art models (FCN, UNet, PSPNet, DeepLabV3+, OCRNet, SETR, SegFormer, HRNet+FCN, and HRNet+OCR), DyHRNet has shown high performance on three public challenging RS image datasets (Vaihingen, Potsdam, and LoveDA). Furthermore, the visual segmentation results, the learned structures, the iteration process analysis, and the ablation study all demonstrate the effectiveness of our proposed model.

BiTSRS: A Bi-Decoder Transformer Segmentor for High-Spatial-Resolution Remote Sensing Images

Semantic segmentation of high-spatial-resolution (HSR) remote sensing (RS) images has been extensively studied, and most of the existing methods are based on convolutional neural network (CNN) models. However, the CNN is regarded to have less power in global representation modeling. In the past few years, methods using transformer have attracted increasing attention and generate improved results in semantic segmentation of natural images, owing to their powerful ability in global information acquisition. Nevertheless, these transformer-based methods exhibit limited performance in semantic segmentation of RS images, probably because of the lack of comprehensive understanding in the feature decoding process. In this paper, a novel transformer-based model named the bi-decoder transformer segmentor for remote sensing (BiTSRS) is proposed, aiming at alleviating the problem of flexible feature decoding, through a bi-decoder design for semantic segmentation of RS images. In the proposed BiTSRS, the Swin transformer is adopted as encoder to take both global and local representations into consideration, and a unique design module (ITM) is designed to deal with the limitation of input size for Swin transformer. Furthermore, BiTSRS adopts a bi-decoder structure consisting of a Dilated-Uper decoder and a fully deformable convolutional network (FDCN) module embedded with focal loss, with which it is capable of decoding a wide range of features and local detail deformations. Both ablation experiments and comparison experiments were conducted on three representative RS images datasets. The ablation analysis demonstrates the contributions of specifically designed modules in the proposed BiTSRS to performance improvement. The comparison experimental results illustrate that the proposed BiTSRS clearly outperforms some state-of-the-art semantic segmentation methods.

CTMU-Net: An Improved U-Net for Semantic Segmentation of Remote-Sensing Images Based on the Combined Attention Mechanism

CTMU-Net: An Improved U-Net for Semantic Segmentation of Remote-Sensing Images Based on the Combined Attention Mechanism

Scale–Relation Joint Decoupling Network for Remote Sensing Image Semantic Segmentation

As we all know, remote sensing (RS) images contain multi-scale and numerous RS objects, along with massive and complex spatial topological relationships, such as the adjacency, proximity relations of same-scale objects, and inclusion relations of cross-scale objects. However, the existing semantic segmentation methods have never explored the cross-scale relations, which are especially important when comes to the situation that the RS objects cannot be accurately identified, they could be supplemented by the surrounding contents. To address the above concern, we propose a scale-relation joint decoupling network (SRJDN) for the semantic segmentation of RS images by simultaneously considering decoupling scales and decoupling relations to excavate more complete relationships of multi-scale RS objects. The SRJDN is performed by following three steps, namely scale decoupling (SD), relation decoupling (RD), and fine-granularity guided fusion (FGF). The SD module uses dilated convolution with different rates to decouple RS objects into different scale feature groups, from small to large scales. Afterward, the RD considers all the spatial topological relationships and decouples these relationships according to the scale, which is divided into two parts, including same-scale relation extraction (SSRE) and cross-scale relation extraction (CSRE). The SSRE establishes the graph structures at each scale independently to mine the relationships of same-scale RS objects and the CSRE constructs the graph in a unified pattern between cross-scales to explore cross-scale target relationships. Third, the FGF module regards small-scale features as fine-granularity representation and applies its attention map to guide the learning of other scale features, which could mine more reliable and comprehensive saliency information and improve the feature consistency. Numerical experiments conducted on two large-scale fine-resolution RS image datasets empirically demonstrate the robustness of the proposed joint decoupling strategy and the effectiveness of the fine-granularity guided fusion in RS image semantic segmentation tasks.

Attention-Based Context Aware Network for Semantic Comprehension of Aerial Scenery.

It is essential for researchers to have a proper interpretation of remote sensing images (RSIs) and precise semantic labeling of their component parts. Although FCN (Fully Convolutional Networks)-like deep convolutional network architectures have been widely applied in the perception of autonomous cars, there are still two challenges in the semantic segmentation of RSIs. The first is to identify details in high-resolution images with complex scenes and to solve the class-mismatch issues; the second is to capture the edge of objects finely without being confused by the surroundings. HRNET has the characteristics of maintaining high-resolution representation by fusing feature information with parallel multi-resolution convolution branches. We adopt HRNET as a backbone and propose to incorporate the Class-Oriented Region Attention Module (CRAM) and Class-Oriented Context Fusion Module (CCFM) to analyze the relationships between classes and patch regions and between classes and local or global pixels, respectively. Thus, the perception capability of the model for the detailed part in the aerial image can be enhanced. We leverage these modules to develop an end-to-end semantic segmentation model for aerial images and validate it on the ISPRS Potsdam and Vaihingen datasets. The experimental results show that our model improves the baseline accuracy and outperforms some commonly used CNN architectures.

Progressive Guidance Edge Perception Network for Semantic Segmentation of Remote-Sensing Images

Remarkable improvements have been seen in the semantic segmentation of remote-sensing images. As an effective structure to aggregate shallow information and deep information, encoder–decoder structure has been widely used in many state-of-the-art models, but it possesses two drawbacks that have not been fully addressed. On the one hand, encoder–decoder structure fuses the features obtained from shallow and deep layers directly; despite harvesting some detailed information, it also brings in noisy features owing to the poor discriminant ability of the shallow layers. On the other hand, existing encoder–decoder structure merely fuses the high-level information generated by the last layer of encoder once, which neglects its guidance ability to the feature aggregation process in the decoder. In this letter, we first propose an edge perception module (EPM) to eliminate the noisy features in the shallow information, as well as enhance features’ structural information. And then, we generate the most suitable guidance information adaptively for different stages in the decoder through high-level information module (HIM). Finally, we apply the guidance information to achieve feature aggregation in the feature aggregation module (FAM). Combined with EPM, HIM, and FAM, our proposed model achieves 89.5% overall accuracy (OA) on the challenging ISPRS Vaihingen test set, which is the new state-of-the-art in the semantic segmentation of remote-sensing images.

Semantic Segmentation of Remote-Sensing Images Based on Multiscale Feature Fusion and Attention Refinement

In recent years, the automatic extraction of remote-sensing image information has attracted full attention. However, the particularity of remote-sensing images and the scarcity of data sets with label information have brought new challenges to existing methods. Therefore, we develop a lightweight semantic segmentation network based onmultiscale feature fusion (MFF) and attention refinement (MFFANet). Our network relies on three crucial modules for improved performance. The multiscale attention refinement module strengthens the representation ability of feature maps extracted by the deep residual network. The MFF module aggregates the information carried by the high-level and low-level features while restoring the image resolution. Furthermore, the boundary enhancement module captures boundary details to solve the semantic ambiguity problem. We achieve 83.5% mean intersection over union (MIoU) on the Urban Semantic 3-D (US3D) data set and 69.3% MIoU on the Vaihingen data set with only 8.2M parameters.

Semantic Segmentation Network of Remote Sensing Images With Dynamic Loss Fusion Strategy

The remote sensing (RS) images are widely used in various industries, among which semantic segmentation of RS images is a common research direction. At the same time, because of the complexity of target information and the high similarity of features between the classes, this task is very challenging. In recent years, semantic segmentation algorithms of RS images have emerged in an endless stream, but most of them are improved around the scale features of the target, and the accuracy has great room for improvement. In this case, we propose a semantic segmentation framework for RS images with dynamic perceptual loss. The framework is improved based on the InceptionV-4 network to form a network that includes contextual semantic fusion and dual-channel atrous spatial pyramid pooling (ASPP). The semantic segmentation network is an encoder-decoder structure. In addition, we design a dynamic perceptual loss module and a dynamic loss fusion strategy by further observing the loss changes of the network, so as to better improve the classified details. Finally, experiment on the ISPRS 2D Semantic Labeling Contest Vaihingen Dataset and Massachusetts Building Dataset. Compared with some segmentation networks, our model has excellent performance.

Learnable Gated Convolutional Neural Network for Semantic Segmentation in Remote-Sensing Images

Semantic segmentation in high-resolution remote-sensing (RS) images is a fundamental task for RS-based urban understanding and planning. However, various types of artificial objects in urban areas make this task quite challenging. Recently, the use of Deep Convolutional Neural Networks (DCNNs) with multiscale information fusion has demonstrated great potential in enhancing performance. Technically, however, existing fusions are usually implemented by summing or concatenating feature maps in a straightforward way. Seldom do works consider the spatial importance for global-to-local context-information aggregation. This paper proposes a Learnable-Gated CNN (L-GCNN) to address this issue. Methodologically, the Taylor expression of the information-entropy function is first parameterized to design the gate function, which is employed to generate pixelwise weights for coarse-to-fine refinement in the L-GCNN. Accordingly, a Parameterized Gate Module (PGM) was designed to achieve this goal. Then, the single PGM and its densely connected extension were embedded into different levels of the encoder in the L-GCNN to help identify the discriminative feature maps at different scales. With the above designs, the L-GCNN is finally organized as a self-cascaded end-to-end architecture that is able to sequentially aggregate context information for fine segmentation. The proposed model was evaluated on two public challenging benchmarks, the ISPRS 2Dsemantic segmentation challenge Potsdam dataset and the Massachusetts building dataset. The experiment results demonstrate that the proposed method exhibited significant improvement compared with several related segmentation networks, including the FCN, SegNet, RefineNet, PSPNet, DeepLab and GSN.For example, on the Potsdam dataset, our method achieved a 93.65% F 1 score and 88.06% I o U score for the segmentation of tiny cars in high-resolution RS images. As a conclusion, the proposed model showed potential for object segmentation from the RS images of buildings, impervious surfaces, low vegetation, trees and cars in urban settings, which largely varies in size and have confusing appearances.

Multilayer semantic segmentation of remote-sensing imagery using a hybrid object-based Markov random field model

ABSTRACTHigh spatial resolution (HR) remote-sensing image usually contains hierarchical semantic information. Many supervised methods have been developed to interpret this information through data training. In this article, without data training, a hybrid object-based Markov random field (HOMRF) model is proposed for multilayer semantic segmentation of remote-sensing images. In this method, label fields of different semantic layers are defined on the same region adjacency graph (RAG) of a given image, and a hybrid framework is suggested to capture and utilize the interactions within and between semantic layers by label fields. Namely a new transition probability matrix is introduced into the energy functions of label fields for describing the semantic context between layers, and the multilevel logistic model is employed to describe the interactions within the same layer. A principled probabilistic inference is developed to determine the optimal solution of the proposed method by iteratively updating each label field until convergence. The computational complexity of the proposed model is , where is the number of classes in all of the layers, is the number of sites in the probability graph of the MRF model, and is the number of iterations. Experimental results from various remote-sensing images demonstrate that the proposed method can produce higher segmentation accuracy than state-of-the-art MRF-based methods.