Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

Abstract

Atmospheric variable prediction (AVP) is crucial for atmospheric and environmental science, as well as practical applications related to production and daily life. Traditional AVP commonly relies on numerical modeling methods. In recent years, deep learning (DL) technologies have introduced new approaches for AVP. With the support of massive high-quality data products, DL methods have achieved remarkable progress. Simultaneously, studying and annotating complex phenomena and processes within vast datasets require specialized researchers and come with substantial costs. The self-supervised learning (SSL) mechanism in DL can facilitate representation learning from unlabeled data, thereby significantly reducing the expenses associated with manual annotation. However, there has been limited exploration of the SSL mechanism in atmospheric variable prediction and analysis. Furthermore, the prevailing large-scale predicting applications based on AVP mainly focus on massive data processing, multi-modal approaches, and foundation model technologies, often neglecting the critical aspects of spatiotemporal representation learning. In light of the above, based on the cutting-edge DL techniques but diverges from the current popular contrastive or generative SSL frameworks, we devise a spatiotemporal (ST) self-supervised predictive learning (SSPL) method (dubbed ST-SSPL) for AVP, which enables the learning of predictive characteristics from unlabeled atmospheric data. Additionally, we improve a multi-group multi-attention (MGMA) method, integrating it into our predictive framework to enhance spatiotemporal representation learning. In experiments conducted on the widely used ERA5 dataset, the proposed ST-SSPL method has achieved superior performance and high efficiency using a simple convolutional neural network (CNN) architecture. These advancements offer crucial methods and references for relevant large-scale applications.