Multimodal Satellite Image Time Series Analysis Using GAN-Based Domain Translation and Matrix Profile

Abstract

The technological development of the remote sensing domain led to the acquisition of satellite image time series (SITS) for Earth Observation (EO) by a variety of sensors. The variability in terms of the characteristics of the satellite sensors requires the existence of algorithms that allow the integration of multiple modalities and the identification of anomalous spatio-temporal evolutions caused by natural hazards. The unsupervised analysis of multimodal SITS proposed in this paper follows a two-step methodology: (i) inter-modality translation and (ii) the identification of anomalies in a change-detection framework. Inter-modality translation is achieved by means of a Generative Adversarial Network (GAN) architecture, whereas, for the identification of anomalies caused by natural hazards, we adapt the task to a similarity search in SITS. In this regard, we provide an extension of the matrix profile concept, which represents an answer to identifying differences and to discovering novelties in time series. Furthermore, the proposed inter-modality translation allows the usage of standard unsupervised clustering approaches (e.g., K-means using the Dynamic Time Warping measure) for mono-modal SITS analysis. The effectiveness of the proposed methodology is shown in two use-case scenarios, namely flooding and landslide events, for which a joint acquisition of Sentinel-1 and Sentinel-2 images is performed.