IEEE GRSS Data Fusion Contest Research Articles

2024 IEEE GRSS Data Fusion Contest: Rapid Flood Mapping [Technical Committees

2024 IEEE GRSS Data Fusion Contest: Rapid Flood Mapping [Technical Committees

2023 IEEE GRSS Data Fusion Contest: Large-Scale Fine-Grained Building Classification for Semantic Urban Reconstruction [Technical Committees

2023 IEEE GRSS Data Fusion Contest: Large-Scale Fine-Grained Building Classification for Semantic Urban Reconstruction [Technical Committees

A novel weakly supervised semantic segmentation framework to improve the resolution of land cover product

Open-source land cover products (LCPs) are essential for many areas of scientific research. However, they have deficiencies such as low accuracy, low resolution, and poor timeliness when applied to a specific area. Therefore, we developed WESUP-LCP, a novel two-stage weakly supervised semantic segmentation framework to improve the resolution of LCPs without any manual labeling cost. In the first stage, we designed a sample transferring module to transfer accurate and representative training samples from low-resolution LCPs. In the second, partially labeled superpixels generated from the transferred labels were used to train the weakly supervised learning network and pseudo-labels were generated through dynamic label propagation during the training process. We designed two experiments to verify the performance of the proposed method, including the experiment on improving the resolution of LCP in specific areas (improving 30 m LCP to 10 m resolution) and the experiment on the public dataset of the multi-temporal semantic change detection task of 2021 IEEE GRSS Data Fusion Contest (improving 30 m LCP to 1 m resolution). The results showed that our method has remarkable advantages over other methods, demonstrating the applicability of WESUP-LCP to improve LCP resolution.

Multimodal Attention-Aware Convolutional Neural Networks for Classification of Hyperspectral and LiDAR Data

The attention mechanism is one of the most influential ideas in the deep learning community, which has shown excellent efficiency in various computer vision tasks. Thus, this article proposes the convolution neural network method with the attention mechanism to enhance the feature extraction of light detection and ranging (LiDAR) data. Meanwhile, our elaborately designed cascaded block contains a short path architecture beneficial for multistage information exchange. With the full exploitation of elevation information from LiDAR data and efficient utilization of the spatial-spectral information underlying hyperspectral data, our method provides a novel solution for multimodal feature fusion. Experiments are conducted on the LiDAR and hyperspectral dataset provided by the 2013 IEEE GRSS Data Fusion Contest and multisource Trento dataset to demonstrate the effectiveness of the proposed method. The experimental results have shown the superior results of the proposed method on both LiDAR and multimodality remote sensing data in comparison with several popular baselines.

Crisscross-Global Vision Transformers Model for Very High Resolution Aerial Image Semantic Segmentation

Semantic segmentation is a key means for understanding very-high resolution (VHR) aerial imagery. With the explosive development of deep learning, deep learning methods are being applied to the segmentation of VHR images, with convolutional neural networks (CNNs) as the basic framework. However, owing to the highly complex details present in VHR images and the high spatial dependence of geographical objects, CNN-based methods are inadequate. This is because the inherent locality of CNNs limits the size of the receptive field, thus limiting the ability to obtain long-range context information. To solve this problem, in this paper, we propose a transformer-based novel deep learning model called crisscross-global vision transformers (CGVT). CGVT exploits the transformer’s inherent ability to obtain long-range context information to solve the restricted receptive field problem. Specifically, we redesign the self-attention mechanism in the transformer and call it crisscross-global attention. It consists of two parts: crisscross transformer encoder block (CC-TEB) and global squeeze transformer encoder block (GS-TEB). CC-TEB overcomes the limitation of the traditional self-attention design (specifically, difficulty applying it to VHR aerial image segmentation) and further increases the local feature representation ability of the model. GS-TEB increases the global feature representation ability of the model. The results of experiments conducted on the popular ISPRS Vaihingen, IEEE GRSS Data Fusion Contest Zeebrugge, and LoveDA Semantic Segmentation Challenge datasets verify the effectiveness and superiority of our proposed method. Specifically, it achieved state-of-the-art performance on both Zeebrugge and LoveDA datasets, and is currently ranked second in Vaihingen dataset.

Beyond single receptive field: A receptive field fusion-and-stratification network for airborne laser scanning point cloud classification

The classification of airborne laser scanning (ALS) point clouds is a critical task of remote sensing and photogrammetry fields. Although recent deep learning-based methods have achieved satisfactory performance, they have ignored the unicity of the receptive field, which makes the ALS point cloud classification remain challenging for the distinguishment of the areas with complex structures and extreme scale variations. In this article, for the objective of configuring multi-receptive field features, we propose a novel receptive field fusion-and-stratification network (RFFS-Net). With a novel dilated graph convolution (DGConv) and its extension annular dilated convolution (ADConv) as basic building blocks, the receptive field fusion process is implemented with the dilated and annular graph fusion (DAGFusion) module, which obtains multi-receptive field feature representation through capturing dilated and annular graphs with various receptive regions. The stratification of the receptive fields with point sets of different resolutions as the calculation bases is performed with Multi-level Decoders nested in RFFS-Net and driven by the multi-level receptive field aggregation loss (MRFALoss) to drive the network to learn in the direction of the supervision labels with different resolutions. With receptive field fusion-and-stratification, RFFS-Net is more adaptable to the classification of regions with complex structures and extreme scale variations in large-scale ALS point clouds. Evaluated on the ISPRS Vaihingen 3D dataset, our RFFS-Net significantly outperforms the baseline (i.e. PointConv) approach by 5.3% on mF1 and 5.4% on mIoU, accomplishing an overall accuracy of 82.1%, an mF1 of 71.6%, and an mIoU of 58.2%. The experiments show that our RFFS-Net achieves a new state-of-the-art classification performance on powerline, car, and fence classes. Furthermore, experiments on the LASDU dataset and the 2019 IEEE-GRSS Data Fusion Contest dataset show that RFFS-Net achieves a new state-of-the-art classification performance. The code is available at github.com/WingkeungM/RFFS-Net.

The 2022 IEEE GRSS Data Fusion Contest: Semisupervised Learning [Technical Committees

Data availability plays a central role in any machine learning setup, especially since the rise of deep learning. Although input data are often available in abundance, reference data used to train and evaluate corresponding approaches are usually scarce due to the high cost of obtaining them. Although this is not limited to remote sensing, it is of particular importance in Earth-observation applications. Semisupervised learning is one approach to mitigate this challenge and leverage the large amount of available input data while relying only on a small, annotated training set.

DGCNN Network Architecture With Densely Connected Point Pairs in Multiscale Local Regions for ALS Point Cloud Classification

Deep learning in 3-D classification tasks focuses on the designs of comprehensive local aggregation operators. Airborne laser scanning (ALS) point clouds have their own characteristics: 1) the object overlaps in the vertical direction; 2) the spatial density is uneven; and 3) the objects present large-scale variations between different categories. However, the dynamic graph convolutional neural network (DGCNN) ignores the inherent properties of ALS point clouds. This study modifies the DGCNN network and proposes a new neural network module called Webconv, which densely connects point pairs in multiscale local regions to learn contextual information. One modified cross entropy loss function with variable weight is proposed to solve the problem of uneven category distributions in ALS points. Because preprocessing such as block partition ignores the context information of the whole region, the conditional random field is used to refine the point cloud. Our approach achieves state-of-the-art performance on the dataset of the 2019 IEEE GRSS Data Fusion Contest 3-D Point Cloud Classification Challenge and can be widely used for ALS point classification.

Report on the 2021 IEEE GRSS Data Fusion Contest—Geospatial Artificial Intelligence for Social Good [Technical Committees

The Image Analysis and Data Fusion Technical Committee (IADF TC) of the IEEE Geoscience and Remote Sensing Society (GRSS) has been organizing the annual Data Fusion Contest (DFC) since 2006. The contest promotes the development of methods for extracting geospatial information from large-scale, multisensor, multimodal, and multitemporal data. It aims to propose new problem settings that are challenging to address with existing techniques and to establish new benchmarks for scientific challenges in remote sensing image analysis <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> – <xref ref-type="bibr" rid="ref2" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> <xref ref-type="bibr" rid="ref3" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> <xref ref-type="bibr" rid="ref4" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> <xref ref-type="bibr" rid="ref5" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[5]</xref> .

IMG2nDSM: Height Estimation from Single Airborne RGB Images with Deep Learning

Estimating the height of buildings and vegetation in single aerial images is a challenging problem. A task-focused Deep Learning (DL) model that combines architectural features from successful DL models (U-NET and Residual Networks) and learns the mapping from a single aerial imagery to a normalized Digital Surface Model (nDSM) was proposed. The model was trained on aerial images whose corresponding DSM and Digital Terrain Models (DTM) were available and was then used to infer the nDSM of images with no elevation information. The model was evaluated with a dataset covering a large area of Manchester, UK, as well as the 2018 IEEE GRSS Data Fusion Contest LiDAR dataset. The results suggest that the proposed DL architecture is suitable for the task and surpasses other state-of-the-art DL approaches by a large margin.

IMAGE TO POINT CLOUD TRANSLATION USING CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR AIRBORNE LIDAR DATA

Abstract. This study introduces a novel image to a 3D point-cloud translation method with a conditional generative adversarial network that creates a large-scale 3D point cloud. This can generate supervised point clouds observed via airborne LiDAR from aerial images. The network is composed of an encoder to produce latent features of input images, generator to translate latent features to fake point clouds, and discriminator to classify false or real point clouds. The encoder is a pre-trained ResNet; to overcome the difficulty of generating 3D point clouds in an outdoor scene, we use a FoldingNet with features from ResNet. After a fixed number of iterations, our generator can produce fake point clouds that correspond to the input image. Experimental results show that our network can learn and generate certain point clouds using the data from the 2018 IEEE GRSS Data Fusion Contest.

Moving Object Tracking via 3-D Total Variation in Remote-Sensing Videos

Tracking moving objects in remote-sensing videos is becoming increasingly important in remote-sensing analysis. This letter presents a novel object tracking method for remote-sensing videos. We start with using the traditional robust principal component analysis (RPCA) model to extract the moving object from the background. To describe the continuity of moving objects in spatial and temporal directions, we incorporate a 3-D total variation (3DTV) regularization into the RPCA model. Considering that the background is not static and the captured videos will contain noise because of the instability of the sensing camera, our proposed method introduces a certain part of the function to model the noise and capture the changes in background. Experimental results on real videos provided by 2016 IEEE GRSS Data Fusion Contest and 2020 Hyperspectral Object Tracking Challenge demonstrate the advantages of the moving object-tracking method via 3-D TV.

The Outcome of the 2021 IEEE GRSS Data Fusion Contest - Track DSE: Detection of Settlements Without Electricity

In this paper, we elaborate on the scientific outcomes of the 2021 Data Fusion Contest (DFC2021), which was organized by the Image Analysis and Data Fusion Technical Committee of the IEEE Geoscience and Remote Sensing Society, on the subject of geospatial artificial intelligence (AI) for social good. The ultimate objective of the contest was to model the state and changes of artificial and natural environments from multimodal and multitemporal remotely sensed data towards sustainable developments. DFC2021 consisted of two challenge tracks: Detection of Settlements without Electricity (DSE) and Multitemporal Semantic change Detection (MSD). We focus here on the outcome of the DSE track. This paper presents the corresponding approaches and reports the results of the best-performing methods during the contest.

Global Land-Cover Mapping With Weak Supervision: Outcome of the 2020 IEEE GRSS Data Fusion Contest

This article presents the scientific outcomes of the 2020 Data Fusion Contest (DFC2020) organized by the Image Analysis and Data Fusion Technical Committee of the IEEE Geoscience and Remote Sensing Society. The 2020 Contest addressed the problem of automatic global land-cover mapping with weak supervision, i.e., estimating high-resolution semantic maps while only low-resolution reference data are available during training. Two separate competitions were organized to assess two different scenarios: 1) high-resolution labels are not available at all; and 2) a small amount of high-resolution labels are available additionally to low-resolution reference data. In this article, we describe the DFC2020 dataset that remains available for further evaluation of corresponding approaches and report the results of the best-performing methods during the contest.

SPATIAL RESOLUTION ENHANCEMENT OF LAND COVER MAPPING USING DEEP CONVOLUTIONAL NETS

Abstract. Multispectral satellite imagery is the primary data source for monitoring land cover change and characterizing land cover at the global scale. However, the accuracy of land cover classification is often constrained by the spatial and temporal resolutions of the acquired satellite images. This paper proposes a novel spatiotemporal fusion method based on deep convolutional neural networks under the application background of massive remote sensing data, as well as the large spatial resolution gaps between MODIS and Sentinel images. The training was taken on the public SEN12MS dataset, while the validation and testing were conducted using ground truth data from the 2020 IEEE GRSS data fusion contest. As a result of data fusion, the synthesized land cover map was more accurate than the corresponding MODIS-derived land cover map, with an enhanced spatial resolution of 10 meters. The ensemble approach can be implemented for improving data quality when generating a global land cover product from coarse satellite imageries.

A geometry-attentional network for ALS point cloud classification

Airborne Laser Scanning (ALS) point cloud classification is a critical task in remote sensing and photogrammetry communities, which can be widely utilized in urban management, powerline surveying and forest monitoring, etc. In particular, the characteristics of ALS point clouds are distinctive in three aspects, (1) numerous geometric instances (e.g. tracts of roofs); (2) extreme scale variations between different categories (e.g. car v.s. roof); (3) discrepancy distribution along the elevation, which should be specifically focused on for ALS point cloud classification. In this paper, we propose a geometry-attentional network consisting of geometry-aware convolution, dense hierarchical architecture and elevation-attention module to embed the three characteristics effectively, which can be trained in an end-to-end manner. Evaluated on the ISPRS Vaihingen 3D Semantic Labeling benchmark, our method achieves the state-of-the-art performance in terms of average F1 score and overall accuracy (OA). Additionally, without retraining, our model trained on the above Vaihingen 3D dataset can also achieve a better result on the dataset of 2019 IEEE GRSS Data Fusion Contest 3D point cloud classification challenge (DFC 3D) than the baseline (i.e. PointSIFT), which verifies the stronger generalization ability of our model.

Multi-Label Learning based Semi-Global Matching Forest

Semi-Global Matching (SGM) approximates a 2D Markov Random Field (MRF) via multiple 1D scanline optimizations, which serves as a good trade-off between accuracy and efficiency in dense matching. Nevertheless, the performance is limited due to the simple summation of the aggregated costs from all 1D scanline optimizations for the final disparity estimation. SGM-Forest improves the performance of SGM by training a random forest to predict the best scanline according to each scanline’s disparity proposal. The disparity estimated by the best scanline acts as reference to adaptively adopt close proposals for further post-processing. However, in many cases more than one scanline is capable of providing a good prediction. Training the random forest with only one scanline labeled may limit or even confuse the learning procedure when other scanlines can offer similar contributions. In this paper, we propose a multi-label classification strategy to further improve SGM-Forest. Each training sample is allowed to be described by multiple labels (or zero label) if more than one (or none) scanline gives a proper prediction. We test the proposed method on stereo matching datasets, from Middlebury, ETH3D, EuroSDR image matching benchmark, and the 2019 IEEE GRSS data fusion contest. The result indicates that under the framework of SGM-Forest, the multi-label strategy outperforms the single-label scheme consistently.

Learning-Shared Cross-Modality Representation Using Multispectral-LiDAR and Hyperspectral Data

Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multi-modal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Lidar and hyperspectral dataset provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.

2019 IEEE GRSS Data Fusion Contest: Large-Scale Semantic 3D Reconstruction [Technical Committees

2019 IEEE GRSS Data Fusion Contest: Large-Scale Semantic 3D Reconstruction [Technical Committees

Fusion of Heterogeneous Earth Observation Data for the Classification of Local Climate Zones

This paper proposes a novel framework for fusing multi-temporal, multispectral satellite images and OpenStreetMap (OSM) data for the classification of local climate zones (LCZs). Feature stacking is the most commonly used method of data fusion but does not consider the heterogeneity of multimodal optical images and OSM data, which becomes its main drawback. The proposed framework processes two data sources separately and then combines them at the model level through two fusion models (the landuse fusion model and building fusion model) that aim to fuse optical images with landuse and buildings layers of OSM data, respectively. In addition, a new approach to detecting building incompleteness of OSM data is proposed. The proposed framework was trained and tested using the data from the 2017 IEEE GRSS Data Fusion Contest and further validated on one additional test (AT) set containing test samples that are manually labeled in Munich and New York. The experimental results have indicated that compared with the feature stacking-based baseline framework, the proposed framework is effective in fusing optical images with OSM data for the classification of LCZs with high generalization capability on a large scale. The classification accuracy of the proposed framework outperforms the baseline framework by more than 6% and 2% while testing on the test set of 2017 IEEE GRSS Data Fusion Contest and the AT set, respectively. In addition, the proposed framework is less sensitive to spectral diversities of optical satellite images and thus achieves more stable classification performance than the state-of-the-art frameworks.