Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network

Shuang Liu,Zhong Zhang,Mei Li,Baihua Xiao,Xiaozhong Cao

doi:10.3390/rs10060822

Abstract

The accurate ground-based cloud classification is a challenging task and still under development. The most current methods are limited to only taking the cloud visual features into consideration, which is not robust to the environmental factors. In this paper, we present the novel joint fusion convolutional neural network (JFCNN) to integrate the multimodal information for ground-based cloud classification. To learn the heterogeneous features (visual features and multimodal features) from the ground-based cloud data, we designed the proposed JFCNN as a two-stream structure which contains the vision subnetwork and multimodal subnetwork. We also proposed a novel layer named joint fusion layer to jointly learn two kinds of cloud features under one framework. After training the proposed JFCNN, we extracted the visual and multimodal features from the two subnetworks and integrated them using a weighted strategy. The proposed JFCNN was validated on the multimodal ground-based cloud (MGC) dataset and achieved remarkable performance, demonstrating its effectiveness for ground-based cloud classification task.

Highlights

Nowadays, many practical applications, such as optical remote sensing application [1], weather prediction [2], precipitation estimation [3] and deep space climate observatory mission [4], require accurate cloud observation techniques
The proposed joint fusion convolutional neural network (JFCNN) is compared with a series of state-of-the-art methods on the multimodal ground-based cloud (MGC) dataset
According to the International cloud classification system criteria published in the World Meteorological Organization (WMO), and the visual similarity in practice, the sky conditions are divided into seven classes: cumulus, cirrus and cirrostratus, cirrocumulus and altocumulus, clear sky, stratocumulus, stratus and altostratus, cumulonimbus and nimbostratus

Summary

Introduction

Many practical applications, such as optical remote sensing application [1], weather prediction [2], precipitation estimation [3] and deep space climate observatory mission [4], require accurate cloud observation techniques. Different measuring instruments have been employed by numerous researchers to obtain the necessary data for cloud classification. The existing ground-based sky imaging devices include whole-sky imager (WSI) [11], total-sky imager (TSI) [12], infrared cloud imager (ICI) [13], all-sky imager (ASI) [14], whole-sky infrared cloud-measuring system (WSIRCMS) [15], etc. They could produce the most available amount of cloud data and, offer researchers an opportunity to understand the cloud conditions better

Methods

Results

Conclusion