Deep tensor fusion network for multimodal ground-based cloud classification in weather station networks

Abstract

Accurate multimodal ground-based cloud classification in weather station networks is a challenging task, because the existing methods fuse cloud visual data and multimodal data at the vector level resulting in the spatial information loss. In this work, we propose a method named deep tensor fusion network (DTFN) for multimodal ground-based cloud classification in weather station networks, which could learn completed cloud information by fusing heterogeneous features at the tensor level in a unified framework. The DTFN is composed of the visual tensor subnetwork (VTN) and the multimodal tensor subnetwork (MTN). The VTN transforms cloud images into cloud visual tensors using a deep network and therefore the spatial information of ground-based cloud images can be maintained. Meanwhile, the MTN is designed as a couple of deconvolutional layers in order to transform the multimodal data into multimodal tensors and ensure the multimodal tensors to be mathematically compatible with cloud visual tensors. Furthermore, to fuse cloud visual tensor and multimodal tensor, we propose the tensor fusion layer to exploit the high-order correlations between them. The DTFN is evaluated on MGCD and exceeds the state-of-the-art methods, which validates its effectiveness for multimodal ground-based cloud classification in weather station networks.