Classification of hyperspectral and LiDAR data by transformer-based enhancement

Abstract

ABSTRACT The integration of multi-modal data allows for a more accurate representation of the ground characteristics. For a comprehensive interpretation of remote sensing data, existing multi-modal data fusion research mainly focuses on the joint utilization of 3D Light Detection and Ranging (LiDAR) and 2D Hyperspectral Image (HSI) data. However, existing algorithms do not pay much attention to the interaction of high-level semantic information between different modal data before fusion. This paper proposes a novel multi-modal data fusion deep learning network with the Cross-Modal Self-Attentive Feature Fusion Transformer (SAFFT). The framework employs a multi-head self-attention layer to fuse various attention information from multiple heads, effectively enhancing advanced feature information from different modalities for comprehensive integration. Experimental results on the Houston 2013 dataset demonstrate the effectiveness of the proposed method, which achieves an overall accuracy (OA) of 94.3757% in classifying 15 semantic classes.