High-resolution Remote Sensing Image Segmentation Research Articles

STIRUnet: SwinTransformer and inverted residual convolution embedding in unet for Sea–Land segmentation

Extraction of coastline from optical remote sensing images is of paramount importance for coastal zone management, erosion monitoring, and intelligent ocean construction. However, nearshore marine environment complexity presents a challenge when capturing small-scale and detailed information regarding coastlines. Furthermore, the presence of numerous tidal flats, suspended sediments, and coastal biological communities exacerbates the reduction in segmentation accuracy, which is particularly noticeable in medium–high-resolution remote sensing image segmentation tasks. Most previous related studies, based primarily on convolutional neural networks (CNNs) or traditional feature extraction methods, faced challenges in detailed pixel-level refinement and lacked comprehensive understanding of the studied images. Therefore, we proposed a new U-shaped deep learning model (STIRUnet) that combines the excellent global modeling ability of SwinTransformer with an improved CNN using an inverted residual module. The proposed method has the capability of global supervised feature learning and layer-by-layer feature extraction, and we conducted sea–land segmentation experiments using GF-HNCD and BSD remote sensing image datasets to validate the performance of the proposed model. The results indicate the following: 1) suspended sediments and coastal biological communities are major contributors to coastline blurring, and 2) the recovery of minute features (e.g., narrow watercourses and microscale artificial structures) effectively enhances edge details and leads to more realistic segmentation outcomes. The findings of this study are highly important in relation of accurate extraction of sea–land information in complex marine environments, and they offer novel insights regarding mixed-pixel identification.

PMNet: a multi-branch and multi-scale semantic segmentation approach to water extraction from high-resolution remote sensing images with edge-cloud computing

In the field of remote sensing image interpretation, automatically extracting water body information from high-resolution images is a key task. However, facing the complex multi-scale features in high-resolution remote sensing images, traditional methods and basic deep convolutional neural networks are difficult to effectively capture the global spatial relationship of the target objects, resulting in incomplete, rough shape and blurred edges of the extracted water body information. Meanwhile, massive image data processing usually leads to computational resource overload and inefficiency. Fortunately, the local data processing capability of edge computing combined with the powerful computational resources of cloud centres can provide timely and efficient computation and storage for high-resolution remote sensing image segmentation. In this regard, this paper proposes PMNet, a lightweight deep learning network for edge-cloud collaboration, which utilises a pipelined multi-step aggregation method to capture image information at different scales and understand the relationships between remote pixels through horizontal and vertical spatial dimensions. Also, it adopts a combination of multiple decoding branches in the decoding stage instead of the traditional single decoding branch. The accuracy of the results is improved while reducing the consumption of system resources. The model obtained F1-score of 90.22 and 88.57 on Landsat-8 and GID remote sensing image datasets with low model complexity, which is better than other semantic segmentation models, highlighting the potential of mobile edge computing in processing massive high-resolution remote sensing image data.

ASPP+-LANet: A Multi-Scale Context Extraction Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Semantic segmentation of remote sensing (RS) images is a pivotal branch in the realm of RS image processing, which plays a significant role in urban planning, building extraction, vegetation extraction, etc. With the continuous advancement of remote sensing technology, the spatial resolution of remote sensing images is progressively improving. This escalation in resolution gives rise to challenges like imbalanced class distributions among ground objects in RS images, the significant variations of ground object scales, as well as the presence of redundant information and noise interference. In this paper, we propose a multi-scale context extraction network, ASPP+-LANet, based on the LANet for semantic segmentation of high-resolution RS images. Firstly, we design an ASPP+ module, expanding upon the ASPP module by incorporating an additional feature extraction channel, redesigning the dilation rates, and introducing the Coordinate Attention (CA) mechanism so that it can effectively improve the segmentation performance of ground object targets at different scales. Secondly, we introduce the Funnel ReLU (FReLU) activation function for enhancing the segmentation effect of slender ground object targets and refining the segmentation edges. The experimental results show that our network model demonstrates superior segmentation performance on both Potsdam and Vaihingen datasets, outperforming other state-of-the-art (SOTA) methods.

Frequency-aware robust multidimensional information fusion framework for remote sensing image segmentation

Urban scene image segmentation is an important research area in high-resolution remote sensing image processing. However, due to its complex three-dimensional structure, interference factors such as occlusion, shadow, intra-class inconsistency, and inter-class indistinction affect segmentation performance. Many methods have combined local and global information using CNNs and Transformers to achieve high performance in remote sensing image segmentation tasks. However, these methods are not stable when dealing with these interference factors. Recent studies have found that semantic segmentation is highly sensitive to frequency information, so we introduced frequency information to make the model learn more comprehensively about different categories of targets from multiple dimensions. By modeling the target with local features, global information, and frequency information, the target features can be learned in multiple dimensions to reduce the impact of interference factors on the model and improve its robustness. In this paper, we consider frequency information in addition to combining CNNs and Transformers for modeling and propose a Multidimensional Information Fusion Network (MIFNet) for high-resolution remote sensing image segmentation of urban scenes. Specifically, we design an information fusion Transformer module that can adaptively associate local features, global semantic information, and frequency information and a relevant semantic aggregation module for aggregating features at different scales to construct the decoder. By aggregating image features at different depths, the specific representation of the target and the correlation between targets can be modeled in multiple dimensions, allowing the network to better recognize and understand the features of each class of targets to resist various interference factors that affect segmentation performance. We conducted extensive ablation experiments and comparative experiments on the ISPRS Vaihingen and ISPRS Potsdam benchmarks to verify our proposed method. In a large number of experiments, our method achieved the best results, with 84.53% and 87.3% mIoU scores on the Vaihingen and Potsdam datasets, respectively, proving the superiority of our method. The source code will be available at https://github.com/JunyuFan/MIFNet.

Research on water extraction from high resolution remote sensing images based on deep learning

Introduction: Monitoring surface water through the extraction of water bodies from high-resolution remote sensing images is of significant importance. With the advancements in deep learning, deep neural networks have been increasingly applied to high-resolution remote sensing image segmentation. However, conventional convolutional models face challenges in water body extraction, including issues like unclear water boundaries and a high number of training parameters.Methods: In this study, we employed the DeeplabV3+ network for water body extraction in high-resolution remote sensing images. However, the traditional DeeplabV3+ network exhibited limitations in segmentation accuracy for high-resolution remote sensing images and incurred high training costs due to a large number of parameters. To address these issues, we made several improvements to the traditional DeeplabV3+ network: Replaced the backbone network with MobileNetV2. Added a Channel Attention (CA) module to the MobileNetV2 feature extraction network. Introduced an Atrous Spatial Pyramid Pooling (ASPP) module. Implemented Focal loss for balanced loss computation.Results: Our proposed method yielded significant enhancements. It not only improved the segmentation accuracy of water bodies in high-resolution remote sensing images but also effectively reduced the number of network parameters and training time. Experimental results on the Water dataset demonstrated superior performance compared to other networks: Outperformed the U-Net network by 3.06% in terms of mean Intersection over Union (mIoU). Outperformed the MACU-Net network by 1.03%. Outperformed the traditional DeeplabV3+ network by 2.05%. The proposed method surpassed not only the traditional DeeplabV3+ but also U-Net, PSP-Net, and MACU-Net networks.Discussion: These results highlight the effectiveness of our modified DeeplabV3+ network with MobileNetV2 backbone, CA module, ASPP module, and Focal loss for water body extraction in high-resolution remote sensing images. The reduction in training time and parameters makes our approach a promising solution for accurate and efficient water body segmentation in remote sensing applications.

Remote Sensing and Deep Learning to Understand Noisy OpenStreetMap

The OpenStreetMap (OSM) project is an open-source, community-based, user-generated street map/data service. It is the most popular project within the state of the art for crowdsourcing. Although geometrical features and tags of annotations in OSM are usually precise (particularly in metropolitan areas), there are instances where volunteer mapping is inaccurate. Despite the appeal of using OSM semantic information with remote sensing images, to train deep learning models, the crowdsourced data quality is inconsistent. High-resolution remote sensing image segmentation is a mature application in many fields, such as urban planning, updated mapping, city sensing, and others. Typically, supervised methods trained with annotated data may learn to anticipate the object location, but misclassification may occur due to noise in training data. This article combines Very High Resolution (VHR) remote sensing data with computer vision methods to deal with noisy OSM. This work deals with OSM misalignment ambiguity (positional inaccuracy) concerning satellite imagery and uses a Convolutional Neural Network (CNN) approach to detect missing buildings in OSM. We propose a translating method to align the OSM vector data with the satellite data. This strategy increases the correlation between the imagery and the building vector data to reduce the noise in OSM data. A series of experiments demonstrate that our approach plays a significant role in (1) resolving the misalignment issue, (2) instance-semantic segmentation of buildings with missing building information in OSM (never labeled or constructed in between image acquisitions), and (3) change detection mapping. The good results of precision (0.96) and recall (0.96) demonstrate the viability of high-resolution satellite imagery and OSM for building detection/change detection using a deep learning approach.

High-Resolution Remote Sensing Image Segmentation Algorithm Based on Improved Feature Extraction and Hybrid Attention Mechanism

Segmentation of high-resolution remote sensing images is one of the hottest topics in deep learning. Compared to ordinary images, high-resolution remote sensing images possess characteristics such as higher intra-class diversity and lower inter-class separability. Additionally, the objects in these images are complex and have smaller sizes. Aiming at the classical segmentation network in remote sensing images, there are some problems, such as inaccurate edge object segmentation, inconsistent segmentation of different types of objects, low detection accuracy, and a high false detection rate. This paper proposes a new hybrid attention model (S-CA), a new coordinate efficient channel attention module (C-ECA), and a new small-target feature extraction network (S-FE). The S-CA model enhances important spatial and channel features in shallow layers, allowing for more detailed feature extraction. The C-ECA model utilizes convolutional layers to capture complex dependencies between variations, thereby better capturing feature information at each position and reducing redundancy in feature channels. The S-FE network can capture the local feature information of different targets more effectively. It enhances the recognition and classification capabilities of various targets and improves the detection rate of small targets. The algorithm is used for segmentation in high-resolution remote sensing images. Experiments were conducted on the public dataset GID-15 based on Gaofen-2 satellite remote sensing images. The experimental results demonstrate that the improved DeepLabV3+ segmentation algorithm for remote sensing images achieved a mean intersection over union (mIoU), mean pixel accuracy (mPA), and mean precision (mP) of 91.6%, 96.1%, and 95.5%, respectively. The improved algorithm is more effective than current mainstream segmentation networks.

FENet: feature enhancement network for land cover classification

ABSTRACT Extracting mask information of buildings and water areas from high resolution remote sensing images is beneficial to monitoring and management of urban development. However, due to different times, different geographical locations and different remote sensing acquisition angles, water areas and buildings will feed back different spectral information. Existing semantic segmentation methods do not pay enough attention to channel information, and the feature information extracted by downsampling is relatively abstract, which is easy to cause the loss of some details in high-resolution images under complex scenes, leading to the misjudgement of buildings and waters. To solve the existing problems, feature enhancement network (FENet) for high-resolution remote sensing image segmentation of buildings and water areas is proposed. By paying more attention to the characteristic information of the passage, the probability of misjudgement of buildings and waters can be reduced and their edge contour information can be enhanced. The self-attention feature module proposed in this paper encodes the context information and transmits it to the local features, and establishes the relationship between channels through the channel feature enhancement module to reduce the loss of channel feature information. The feature fusion module fuses feature information of different scales in space and outputs more detailed prediction images. Comparative experiments show that this model is superior to the existing classical semantic segmentation model. Compared with the existing models, the proposed method can achieve 2% improvement than PSPNet on the indicator MIoU, and the final MIoU reaches 82.85% for land cover dataset. This study demonstrates the advantages of our proposed method in land cover classification and detection.

Fuzzy neighbourhood neural network for high-resolution remote sensing image segmentation

ABSTRACT Remote sensing image segmentation plays an important role in many industrial-grade image processing applications. However, the problem of uncertainty caused by intraclass heterogeneity and interclass blurring is prevalent in high-resolution remote sensing images. Moreover, the complexity of information in high-resolution remote sensing images leads to a large amount of background information around objects. To solve this problem, a new fuzzy convolutional neural network is proposed in this paper. This network resolves the ambiguity and uncertainty of feature information by introducing a fuzzy neighbourhood module in the deep learning network structure. In addition, it adds a multi-attention gating module to highlight small object features and separate them from the complex background information to achieve fine segmentation of high-resolution remote sensing images. Experimental results on three different segmentation datasets suggest that the proposed method has higher segmentation accuracy and better performance than other deep learning networks, especially for complicated shadow information. Code will be provided in (https://github.com/tingtingqu/code).

High-resolution remote sensing image segmentation based on dual-modal efficient feature learning

High-resolution remote sensing image segmentation based on dual-modal efficient feature learning

Semi-supervised Deep Learning via Transformation Consistency Regularization for Remote Sensing Image Semantic Segmentation

Deep convolutional neural networks (CNNs) have gotten a lot of press in the last several years, especially in domains like computer vision (CV) and remote sensing (RS). However, achieving superior performance with deep networks highly depends on a massive amount of accurately labeled training samples. In real-world applications, gathering a large number of labeled samples is time-consuming and labor-intensive, especially for pixel-level data annotation. This dearth of labels in land-cover classification is especially pressing in the RS domain because high-precision, high-quality labeled samples are extremely difficult to acquire, but unlabeled data is readily available. In this study, we offer a new semi-supervised deep semantic labeling framework for semantic segmentation of high-resolution RS images to take advantage of the limited amount of labeled examples and numerous unlabeled samples. Our model uses transformation consistency regularization (TCR) to encourage consistent network predictions under different random transformations or perturbations. We try three different transforms to compute the consistency loss and analyze their performance. Then, we present a deep semi-supervised semantic labeling technique by using a hybrid transformation consistency regularization (HTCR). A weighted sum of losses, which contains a supervised term computed on labeled samples and an unsupervised regularization term computed on unlabeled data, may be used to update the network parameters in our technique. Our comprehensive experiments on two RS datasets confirmed that the suggested approach utilized latent information from unlabeled samples to obtain more precise predictions and outperformed existing semi-supervised algorithms in terms of performance. Our experiments further demonstrated that our semi-supervised semantic labeling strategy has the potential to partially tackle the problem of limited labeled samples for high-resolution RS image land-cover segmentation.

Land Use Classification Using Improved U-Net in Remote Sensing Images of Urban and Rural Planning Monitoring

Aiming at the problem of low accuracy and efficiency of existing land use classification methods for high-resolution remote sensing image segmentation, a land use classification method using improved U-Net in remote sensing images of urban and rural planning monitoring is proposed. First, taking the high-resolution remote sensing images of different remote sensing satellites as the data source, the remote sensing images in the data source are registered and cropped so that the pixels at the corresponding positions represent the same geographical location. Then, the encoder of the U-Net model is combined with the residual module to share the network parameters and avoid the degradation of the deep network. The dense connection module is integrated into the decoder to connect the shallow features with the deep features, so as to obtain new features and improve the feature utilization rate. Finally, the depthwise separable convolution is used to process the spatial and channel information of the convolution process separately to reduce the model parameters. Experiments show that the pixel accuracy, recall rate, precision rate, and average intersection-over-union ratio of the proposed land use classification method based on improved U-Net are 92.35%, 80.56%, 83.45%, and 86.75%, respectively, which are better than the compared methods. Therefore, the proposed method is proved to have good land use classification ability.

High-Resolution Remote Sensing Image Segmentation Framework Based on Attention Mechanism and Adaptive Weighting

Semantic segmentation has been widely used in the basic task of extracting information from images. Despite this progress, there are still two challenges: (1) it is difficult for a single-size receptive field to acquire sufficiently strong representational features, and (2) the traditional encoder-decoder structure directly integrates the shallow features with the deep features. However, due to the small number of network layers that shallow features pass through, the feature representation ability is weak, and noise information will be introduced to affect the segmentation performance. In this paper, an Adaptive Multi-Scale Module (AMSM) and Adaptive Fuse Module (AFM) are proposed to solve these two problems. AMSM adopts the idea of channel and spatial attention and adaptively fuses three-channel branches by setting branching structures with different void rates, and flexibly generates weights according to the content of the image. AFM uses deep feature maps to filter shallow feature maps and obtains the weight of deep and shallow feature maps to filter noise information in shallow feature maps effectively. Based on these two symmetrical modules, we have carried out extensive experiments. On the ISPRS Vaihingen dataset, the F1-score and Overall Accuracy (OA) reached 86.79% and 88.35%, respectively.

A Multi-Level Feature Fusion Network for Remote Sensing Image Segmentation.

High-resolution remote sensing image segmentation is a mature application in many industrial-level image applications and it also has military and civil applications. The scene analysis needs to be automated as much as possible with high-resolution remote sensing images. This plays a significant role in environmental disaster monitoring, forestry industry, agricultural farming, urban planning, and road analysis. This study proposes a multi-level feature fusion network (MFNet) that can integrate the multi-level features in the backbone to obtain different types of image information. Finally, the experiments in this study demonstrate that the proposed network can achieve good segmentation results in the Vaihingen and Potsdam datasets. By aiming to achieve a large difference in the scale of the target objects in remote sensing images and achieving a poor recognition result for small objects, a multi-level feature fusion solution is proposed in this study. This investigation improves the recognition results of the remote sensing image segmentation to a certain extent.

DenseNet-Based Land Cover Classification Network With Deep Fusion

Recently, fully convolutional network (FCN)-based (Long <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , 2015) networks have made impressive success in semantic segmentation, and these approaches achieve satisfactory results in natural images. However, in the field of high-resolution remote sensing image segmentation, the accuracy has a considerable huge gap compared with that of natural images. Through the development process of semantic segmentation, we found that the key to accurate segmentation is the context. Effective networks can always obtain large contexts, which means that context is the key to one successful segmentation network. For high-resolution remote sensing images, their elements always extend to large scope and they have no clear or regular boundaries. As a result, it needs more context to correctly classify each pixel. However, the networks designed for natural images obviously do not meet this requirement, and thus, they achieve poor segmentation results for high-resolution images. Therefore, we do some targeted improvements. Based on one powerful backbone, we add two new fusions called unit fusion and cross-level fusion, respectively. Unit fusion makes the connection from the encoder part to the decoder part not only occur in the final output of each dense block but also in the middle feature layers inside one dense block. These added fusions make feature fusion in the same level more complete, which is of great significance for complex and zigzag boundary areas. As a complement for unit fusion, cross-level fusion aims to enhance the fusion of different dense blocks. Specifically, cross-level fusion learns from the internal structure of the dense block and applies the design to the whole network level. It can incorporate nonadjacent features and rapidly increase the receptive field and context, which is very effective for the segmentation of targets with very large sizes. Experiments on Deepglobe (Demir <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , 2018) show significant improvements in our work.

Flexible Hierarchical Gaussian Mixture Model for High-Resolution Remote Sensing Image Segmentation

The Gaussian mixture model (GMM) plays an important role in image segmentation, but the difficulty of GMM for modeling asymmetric, heavy-tailed, or multimodal distributions of pixel intensities significantly limits its application. One effective way to improve the segmentation accuracy is to accurately model the statistical distributions of pixel intensities. In this study, an innovative high-resolution remote sensing image segmentation algorithm is proposed based on a flexible hierarchical GMM (HGMM). The components are first defined by the weighted sums of elements, in order to accurately model the complicated distributions of pixel intensities in object regions. The elements of components are defined by Gaussian distributions to model the distributions of pixel intensities in local regions of the object region. Following the Bayesian theorem, the segmentation model is then built by combining the HGMM and the prior distributions of parameters. Finally, a novel birth or death Markov chain Monte Carlo (BDMCMC) is designed to simulate the segmentation model, which can automatically determine the number of elements and flexibly model complex distributions of pixel intensities. Experiments were implemented on simulated and real high-resolution remote sensing images. The results show that the proposed algorithm is able to flexibly model the complicated distributions and accurately segment images.

Another look on region merging procedure from seed region shift for high-resolution remote sensing image segmentation

Region merging method is widely used for remote sensing image segmentation in Geographic Object-Based Image Analysis (GEOBIA) because of its simplicity and effectiveness. Instead of improving the merging strategy, similarity measure, and stopping rule for region merging method as usual, we aim at exploring the effectiveness of the seed region shift on region merging-based segmentation. Different region merging procedures with different seed region shift frequencies are compared by fixing other conditions, demonstrating that the shift of seed regions serves as one of the key impacts to segmentation accuracy for region merging method. If the seed regions keep fixed during region merging procedure, it will lead to uneven expansion of regions and consequently low segmentation accuracy. However, if the seed regions can be dynamically shifted during region merging procedure, it will lead to even expansion of regions and achieve similar segmentation performance for different region merging strategies. The findings could be beneficial to selecting or further improving image segmentation method for GEOBIA.

Problems of encoder-decoder frameworks for high-resolution remote sensing image segmentation: Structural stereotype and insufficient learning

This work explores the use of deep convolutional neural networks for high resolution remote sensing imagery segmentation. Encoder-decoder frameworks are popular in semantic image segmentation. However, encoder-decoder models face two main problems. The one is structural stereotype which is receptive fields imbalance rooted in this kind of frameworks. The other is insufficient learning that deeper neural networks tend to encounter the notorious problem of vanishing gradients. Structural stereotype leads to unfair learning and inhomogeneous reasoning. We are the first to reveal the problem and propose ensemble training and inference strategies to suppress the adverse consequences of structural stereotype as far as possible. To alleviate the problem of insufficient learning, we propose a novel residual architecture for encoder-decoder models. The proposed method yields state-of-the-art performances on the ISPRS 2D semantic labeling contest benchmark.

Towards Operational Satellite-Based Damage-Mapping Using U-Net Convolutional Network: A Case Study of 2011 Tohoku Earthquake-Tsunami

The satellite remote-sensing-based damage-mapping technique has played an indispensable role in rapid disaster response practice, whereas the current disaster response practice remains subject to the low damage assessment accuracy and lag in timeliness, which dramatically reduces the significance and feasibility of extending the present method to practical operational applications. Therefore, a highly efficient and intelligent remote-sensing image-processing framework is urgently required to mitigate these challenges. In this article, a deep learning algorithm for the semantic segmentation of high-resolution remote-sensing images using the U-net convolutional network was proposed to map the damage rapidly. The algorithm was implemented within a Microsoft Cognitive Toolkit framework in the GeoAI platform provided by Microsoft. The study takes the 2011 Tohoku Earthquake-Tsunami as a case study, for which the pre- and post-disaster high-resolution WorldView-2 image is used. The performance of the proposed U-net model is compared with that of deep residual U-net. The comparison highlights the superiority U-net for tsunami damage mapping in this work. Our proposed method achieves the overall accuracy of 70.9% in classifying the damage into “washed away,” “collapsed,” and “survived” at the pixel level. In future disaster scenarios, our proposed model can generate the damage map in approximately 2–15 min when the preprocessed remote-sensing datasets are available. Our proposed damage-mapping framework has significantly improved the application value in operational disaster response practice by substantially reducing the manual operation steps required in the actual disaster response. Besides, the proposed framework is highly flexible to extend to other scenarios and various disaster types, which can accelerate operational disaster response practice.

An unsupervised multi-scale segmentation method based on automated parameterization

The continuous improvement of the launched satellites’ spatial and spectral resolutions has brought new challenges for remote sensing image segmentation technology. The traditional supervised methods greatly depend on artificial interpretation and reduce the degree of automation and robustness of image segmentation. Therefore, the article proposes a novel unsupervised multi-scale segmentation method for high-resolution remote sensing images based on automated parameterization and it mainly includes three steps, adaptive selection of scale parameter (SP) based on local area homogeneity index J-value, multi-scale segmentation based on the inter-scales boundaries constraint strategy, and region merging based on multi-features. The article makes experiments by multi-group high-resolution remote sensing images of different launched satellites and compares the proposed method with the well-known commercial software eCognition and a traditional supervised method. The results show that the proposed method can locate the object edges more accurately and extract the object outlines more completely, and needs no human intervention in segmentation process, so it can provide a generic and effective unsupervised solution for high-resolution remote sensing image segmentation.