Compact Cloud Detection with Bidirectional Self-Attention Knowledge Distillation

Yajie Chai,Lei Wang,Wenhui Diao,Kun Fu,Yingchao Feng,Zhiyuan Yan,Xian Sun

doi:10.3390/rs12172770

Abstract

The deep convolutional neural network has made significant progress in cloud detection. However, the compromise between having a compact model and high accuracy has always been a challenging task in cloud detection for large-scale remote sensing imagery. A promising method to tackle this problem is knowledge distillation, which usually lets the compact model mimic the cumbersome model’s output to get better generalization. However, vanilla knowledge distillation methods cannot properly distill the characteristics of clouds in remote sensing images. In this paper, we propose a novel self-attention knowledge distillation approach for compact and accurate cloud detection, named Bidirectional Self-Attention Distillation (Bi-SAD). Bi-SAD lets a model learn from itself without adding additional parameters or supervision. With bidirectional layer-wise features learning, the model can get a better representation of the cloud’s textural information and semantic information, so that the cloud’s boundaries become more detailed and the predictions become more reliable. Experiments on a dataset acquired by GaoFen-1 satellite show that our Bi-SAD has a great balance between compactness and accuracy, and outperforms vanilla distillation methods. Compared with state-of-the-art cloud detection models, the parameter size and FLOPs are reduced by 100 times and 400 times, respectively, with a small drop in accuracy.

Highlights

With the rapid development of remote sensing technology, many remote sensing images (RSIs) with high resolution can be obtained and have been widely used in the fields of resource survey, disaster prevention, environmental pollution monitoring, urbanization studies, etc. [1,2]
We propose a novel bidirectional self-attention distillation (Bi-SAD) method for compact and accurate cloud detection
We find that after training with our Bidirectional Self-Attention Distillation (Bi-SAD), the student model almost reaches the same performance as the teacher model and even better on mean intersection over union (mIoU) and F1, and our Bi-SAD outperforms the state-of-the-art distillation methods, which proves that our Bi-SAD is more powerful in cloud detection

Summary

Introduction

With the rapid development of remote sensing technology, many remote sensing images (RSIs) with high resolution can be obtained and have been widely used in the fields of resource survey, disaster prevention, environmental pollution monitoring, urbanization studies, etc. [1,2]. Thresholding based methods [4,5,6,7,8,9,10] and machine learning based methods [11,12,13,14,15,16,17,18] are widely used in cloud detection, because they are simple, effective, and fast in calculation. Machine learning-based methods have distinct advantages in extracting more robust high-level features from images and improving the detection performance. They heavily rely on the manually designed features, which requires sufficient prior knowledge and makes it difficult to accurately capture the cloud features in the complex environment. Inner-SAD gives the model a better ability to distinguish between snow/ice and clouds.

Methods

Results

Discussion

Conclusion