Multilevel Spatial-Channel Feature Fusion Network for Urban Village Classification by Fusing Satellite and Streetview Images

Abstract

Urban Villages (UV) refer to areas of urban informal settlements lagging behind the rapid urbanization process. Recent studies focus on using satellite images to classify UV. However, satellite images only capture objects from a bird-eye perspective, thus cannot obtain complex spatial relationships between objects. In UV areas, buildings and objects are usually dense, small in size, and obscure each other. Therefore, it is challenging to classify UV accurately using only satellite images with bird-eye perspectives. In this paper, to solve this problem, we proposed a novel method that uses satellite images combined with streetview images to classify UV. Specifically, we propose a novel multilevel spatial-channel feature fusion network, namely FusionMixer, that integrates CNN-based feature extraction modules and a multilevel spatial-channel feature fusing layer to make an optimal UV classification. Experiments were conducted in Shenzhen City (the RsSt-ShenzhenUV dataset) and a public UV dataset (the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S²UV</i> dataset). The proposed FusionMixer achieved an increase of OA by 8.83% and 8.84%, and improves Kappa by 0.1765 and 0.1770 in the validation set and testing set, compared to the second-best fusion models in RsSt-ShenzhenUV dataset. Experiments in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S²UV</i> dataset show that the proposed FusionMixer improves OA by 1.82% and Kappa by 0.04 compared to other methods. We also added a set of experiments on a public dataset (Houston dataset) and compare our method with the current state-of-the-art multimodal fusion methods to prove the generalization of the proposed FusionMixer in fusing other multimodality data. These experiments confirmed the superior performance of the proposed FusionMixer.