Fully‐connected semantic segmentation of hyperspectral and LiDAR data

Abstract

Semantic segmentation is an emerging field in the computer vision community where one can segment and label an object all at once, by considering the effects of the neighbouring pixels. In this study, the authors propose a new semantic segmentation model that fuses hyperspectral images with light detection and ranging (LiDAR) data in the three‐dimensional space defined by Universal Transverse Mercator (UTM) coordinates and solves the task using a fully‐connected conditional random field (CRF). First, the authors’ pairwise energy in the CRF model takes into account the UTM coordinates of the data; and performs fusion in the real world coordinates. Second, as opposed to the commonly used Markov random fields (MRFs) which consider only the nearby pixels; the fully‐connected CRF considers all the pixels in an image to be connected. In doing so, they show that these long‐term interactions significantly enhance the results when compared to traditional MRF models. Third, they propose an adaptive scaling scheme to decide the weights of LiDAR and hyperspectral sensors in shadowy or sunny regions. Experimental results on the Houston dataset indicate the effectiveness of their method in comparison to the several MRF based approaches as well as other competing methods.