Deep multimodal fusion for ground-based cloud classification in weather station networks

Shuang Liu,Mei Li

doi:10.1186/s13638-018-1062-0

Abstract

Most existing methods only utilize the visual sensors for ground-based cloud classification, which neglects other important characteristics of cloud. In this paper, we utilize the multimodal information collected from weather station networks for ground-based cloud classification and propose a novel method named deep multimodal fusion (DMF). In order to learn the visual features, we train a convolutional neural network (CNN) model to obtain the sum convolutional map (SCM) by using a pooling operation across all the feature maps in deep layers. Afterwards, we employ a weighted strategy to integrate the visual features with multimodal features. We validate the effectiveness of the proposed DMF on the multimodal ground-based cloud (MGC) dataset, and the experimental results demonstrate the proposed DMF achieves better results than the state-of-the-art methods.

Highlights

Clouds, as one of the major meteorological phenomena, play a profound role in climate predictions and services [1, 2]
3 Experimental results we conduct a series of experiments on the multimodal ground-based cloud (MGC) dataset to evaluate the effectiveness of the proposed deep multimodal fusion (DMF)
According to the International cloud classification system criteria published in the World Meteorological Organization (WMO), considering the visual similarity in practice, the sky conditions are divided into seven classes, i.e., cumulus, cirrus, altocumulus, clear sky, stratus, stratocumulus, and cumulonimbus

Summary

Introduction

As one of the major meteorological phenomena, play a profound role in climate predictions and services [1, 2]. One trend is to develop the ground-based sky imagers such as whole-sky imager (WSI) [4], total-sky imager (TSI) [5], infrared cloud imager (ICI) [6], all-sky imager (ASI) [7, 8], whole-sky infrared cloud measuring system (WSIRCMS) [9], and day/night whole sky imagers (D/N WSIs) [10]. Benefiting from these devices, a number of ground-based cloud images are available for developing automatic classification algorithms. Calbo et al [1] extracted statistical

Methods

Results

Conclusion