Classification Of Image Time Series Research Articles

A constrastive semi-supervised deep learning framework for land cover classification of satellite time series with limited labels

In this work, we present a new semi-supervised learning framework to cope with satellite image time series (SITS) classification in a data paucity scenario, considering extreme low levels of supervision. The proposed methodology, referred as S3ITS (Semi-Supervised Satellite Image Time Series classification method), is based on temporal convolutional neural networks and it takes advantage of both labelled and unlabelled information. S3ITS enforces the data to be projected in a discriminative manifold via contrastive learning, in order to produce a data representation where samples belonging to the same category are closer than the ones belonging to different ones. Pseudo-labelling is employed on unlabelled samples to take the most out of the available unlabelled information. Experiments on two study sites described by SITS of Sentinel-2 images highlight the quality of the proposed method with respect to common SITS-based classification methods and recent machine learning approaches especially tailored for the semi-supervised classification of multi-variate time series data.

Astronomical image time series classification using CONVolutional attENTION (ConvEntion)

Aims. The treatment of astronomical image time series has won increasing attention in recent years. Indeed, numerous surveys following up on transient objects are in progress or under construction, such as the Vera Rubin Observatory Legacy Survey for Space and Time (LSST), which is poised to produce huge amounts of these time series. The associated scientific topics are extensive, ranging from the study of objects in our galaxy to the observation of the most distant supernovae for measuring the expansion of the universe. With such a large amount of data available, the need for robust automatic tools to detect and classify celestial objects is growing steadily. Methods. This study is based on the assumption that astronomical images contain more information than light curves. In this paper, we propose a novel approach based on deep learning for classifying different types of space objects directly using images. We named our approach ConvEntion, which stands for CONVolutional attENTION. It is based on convolutions and transformers, which are new approaches for the treatment of astronomical image time series. Our solution integrates spatio-temporal features and can be applied to various types of image datasets with any number of bands. Results. In this work, we solved various problems the datasets tend to suffer from and we present new results for classifications using astronomical image time series with an increase in accuracy of 13%, compared to state-of-the-art approaches that use image time series, and a 12% increase, compared to approaches that use light curves.

Attention to Both Global and Local Features: A Novel Temporal Encoder for Satellite Image Time Series Classification

Satellite image time series (SITS) classification is a challenging application concurrently driven by long-term, large-scale, and high spatial-resolution observations acquired by remote sensing satellites. The focus of current SITS classification research is to exploit the richness of temporal information in SITS data. In the literature, self-attention mechanism-based networks, which are capable of capturing global temporal attention, have achieved state-of-the-art results in SITS classification. However, these methods lack attention to local temporal information, which is also significant for SITS classification tasks. To explore the potential of different scales of temporal information in SITS data, a global–local temporal attention encoder (GL-TAE) is proposed in this paper. GL-TAE has two submodules set up in parallel, one of which is a lightweight temporal attention encoder (LTAE) for extracting global temporal attention and the other is lightweight convolution (LConv) for extracting local temporal attention. Compared with methods exploring global-only or local-only temporal features, the proposed GL-TAE can achieve better performance on two public SITS datasets, which proves the effectiveness of hybrid global–local temporal attention features. The experiments also demonstrate that GL-TAE is a lightweight model, which achieves the same performance as other models but with fewer parameters.

Channel Attention-Based Temporal Convolutional Network for Satellite Image Time Series Classification

Satellite image time series classification has become a research focus with the launch of new remote sensing sensors capable of capturing images with high spatial, spectral, and temporal resolutions. In particular, in the field of crop classification, time dimension information is particularly important. Although some advanced machine learning algorithms, such as random forests (RFs), can achieve good results, they often ignore the time series information. To make full use of temporal and spectral information in multitemporal remote sensing images, a channel attention-based temporal convolutional network (CA-TCN) is proposed in this letter. Specifically, the proposed method is composed of two main modules: temporal convolutional network and attention block. The temporal convolutional network can capture long-range dependence by using a hierarchy of temporal convolutional filters. To capture relevant information inside the sequence and enhance the important information, the attention block is used to enhance the important features in the channel dimension since not all bands contain equal information in crop type classification. The proposed CA-TCN can excavate deeper phenological characteristics. Compared to the temporal attention-based temporal convolutional network and other deep learning-based models, the proposed CA-TCN has achieved state-of-the-art performance in the Breizhcrops dataset with fewer parameters.

Influence of Data Representations and Deep Architectures in Image Time Series Classification

Image time series, such as Satellite Image Time Series (SITS) or MRI functional sequences in the medical domain, carry both spatial and temporal information. In many pattern recognition applications such as image classification, taking into account such rich information may be crucial and discriminative during the decision making stage. However, the extraction of spatio-temporal features from image time series is difficult to handle due to the complex representation of the data cube. In this paper, we present a strategy based on Random Walk to build a novel segment-based representation of the data, passing from a 2D[Formula: see text] dimension to a 2D one, more easily manipulable and without losing too much spatial information. Such new representation is then used to feed a classical Convolutional Neural Network (CNN) in order to learn spatio-temporal features with only 2D convolutions and to classify image time series data for a particular classification problem. The influence of the way the 2D[Formula: see text] data are represented, as well as the impact of the network architectures on the results, are carefully studied. The interest of this approach is highlighted on a remote sensing application for the classification of complex agricultural crops.

Deep-STaR: Classification of image time series based on spatio-temporal representations

Image time series (ITS) represent complex 3D (2D+t in practice) data that are now daily produced in various domains, from medical imaging to remote sensing. They contain rich spatio-temporal information allowing the observation of the evolution of a sensed scene over time. In this work, we focus on the classification task of ITS, as often available in remote sensing tasks. An underlying problem here is to consider jointly the spatial and the temporal dimensions of the data. We present Deep-STaR, a method to learn such features from ITS data to proceed to their classification. Instead of reasoning in the original 2D+t space, we investigate novel 2D planar data representations, containing both temporal and spatial information. Such representations are a novel way to structure the ITS, compatible with deep learning architectures. They are used to feed a convolutional neural network to learn spatio-temporal features with 2D convolutions, leading ultimately to classification decision. To enhance the explainability of the results, we also propose a post-hoc attention mechanism, enabled by this new approach, providing a semantic map giving some insights for the taken decision. Deep-STaR is evaluated on a remote sensing application, for the classification of agricultural crops from satellite ITS. The results highlight the benefice of this method, compared to the literature, and its interest to make easier the interpretation of ITS to understand spatio-temporal phenomena.

Cost-Sensitive Self-Paced Learning With Adaptive Regularization for Classification of Image Time Series

The classification of image time series has potential significance in the field of land-cover analysis with the increasing number of remote sensing images. The key problem of the classification of image time series is how to transfer the already available knowledge on the source domain to the target domain. Nevertheless, most of the existing methods do not consider the impact of different sample costs on the classifier during transferring. In addition, it is very difficult to collect reliable labeled samples with changed or unchanged categories between the source domain and the target domain in the case of a large number of training samples. In order to alleviate the above problems, we propose a cost-sensitive self-paced learning (CSSPL) framework with adaptive regularization for the classification of image time series in this paper. Considering that the costs of different samples cannot be completely equal to the classifier, different cost values are assigned to each type of error first, then we minimize the total cost to give the change detection classifier a preference on the unchanged class, aiming to reduce wrong label propagation from the source to the target image. Besides, an adaptive mixture weight regularizer is designed to automatically assign sample weight based on the loss value of the data set, which enables more reliable sample weights to be selected for training. Experimental results show that the proposed algorithm provides a set of reliable samples for the training of classifier and achieves a promising improvement on classification accuracy.

Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification

Satellite image time series (SITS) classification is a major research topic in remote sensing and is relevant for a wide range of applications. Deep learning approaches have been commonly employed for the SITS classification and have provided state-of-the-art performance. However, deep learning methods suffer from overfitting when labeled data are scarce. To address this problem, we propose a novel self-supervised pretraining scheme to initialize a transformer-based network by utilizing large-scale unlabeled data. In detail, the model is asked to predict randomly contaminated observations given an entire time series of a pixel. The main idea of our proposal is to leverage the inherent temporal structure of satellite time series to learn general-purpose spectral-temporal representations related to land cover semantics. Once pretraining is completed, the pretrained network can be further adapted to various SITS classification tasks by fine-tuning all the model parameters on small-scale task-related labeled data. In this way, the general knowledge and representations about SITS can be transferred to a label-scarce task, thereby improving the generalization performance of the model as well as reducing the risk of overfitting. Comprehensive experiments have been carried out on three benchmark datasets over large study areas. Experimental results demonstrate the effectiveness of the proposed pretraining scheme, leading to substantial improvements in classification accuracy using transformer, 1-D convolutional neural network, and bidirectional long short-term memory network. The code and the pretrained model will be available at https://github.com/linlei1214/SITS-BERT upon publication.

Simple Nonlinear Iterative Temporal Clustering

Classifying dense satellite image time series has become a necessity, especially with the recent efforts to create analysis ready data cubes. Approaches developed to perform this task are usually pixel-based. Even though these approaches can achieve good results, they do not take advantage of the intrinsic spatial correlation of geographic data nor do they consider spatial heterogeneity along with the time series. Region-based classification is a suitable solution to incorporate contextual information for dense satellite image time series classification. In this article, we introduce a new segmentation method based on a superpixel approach. This method creates multitemporal superpixels, which are meaningful regions in space and time. To evaluate the performance of the proposed method, tests were performed on two data sets using a total of 23 ground-truth references. Experimental results showed that the method performed well, achieving a good boundary agreement and obtaining high scores on the three metrics used for evaluation.

Extreme anomaly event detection in biosphere using linear regression and a spatiotemporal MRF model

Detecting abnormal events within time series is crucial for analyzing and understanding the dynamics of the system in many research areas. In this paper, we propose a methodology to detect these anomalies in multivariate environmental data. Five biosphere variables from a preliminary version of the Earth System Data Cube have been used in this study: Gross Primary Productivity, Latent Energy, Net Ecosystem Exchange, Sensible Heat and Terrestrial Ecosystem Respiration. To tackle the spatiotemporal dependencies of the biosphere variables, the proposed methodology after preprocessing the data is divided into two steps: a feature extraction step applied to each time series in the grid independently, followed by a spatiotemporal event detection step applied to the obtained novelty scores over the entire study area. The first step is based on the assumption that the time series of each variable can be represented by an autoregressive moving average (ARMA) process, and the anomalies are those time instances that are not well represented by the estimated ARMA model. The Mahalanobis distance of the ARMA models’ multivariate residuals is used as a novelty score. In the second step, the obtained novelty scores of the entire study are treated as time series of images. Markov random fields (MRFs) provide an effective and theoretically well-established methodology for integrating spatiotemporal dependency into the classification of image time series. In this study, the classification of the novelty score images into three classes, intense anomaly, possible anomaly, and normal, is performed using unsupervised K-means clustering followed by multi-temporal MRF segmentation applied recursively on the images of each consecutive L ge 1 time steps. The proposed methodology was applied to an area covering Europe and Africa. Experimental results and validation based on known historic events show that the method is able to detect historic events and also provides a useful tool to define sensitive regions.

Consistent classification of image time series with automatic adaptive signature generalization

Long-term data archives such as Landsat offer the potential for understanding land cover dynamics over large areas, but limited progress has been made towards realizing this potential due to data availability and computational limitations. Those limitations are less relevant now, and there is renewed interest in developing reliable methods of automatically and consistently classifying time series of remotely sensed images. Our objective was to develop a method of automatically classifying temporally irregular time series (i.e., non-anniversary date images in consecutive years) of images with a minimum of parameterization and a priori information. In contrast to traditional signature extension methods, the automatic adaptive signature generalization procedure (AASG) adapts class spectral signatures to individual images and therefore requires no image correction procedure. Class signatures are derived from pixels with stable land cover through time. We tested the performance of AASG relative to traditional signature extension with various image corrections, and explored the sensitivity of AASG to a thresholding parameter (c) controlling stable site identification. AASG performed as well as signature extension with atmospheric correction (κ=0.68), and better than signature extension with relative (κ=0.65) and TOA reflectance (κ=0.56) image corrections for a summer–summer image pair. Additionally, we demonstrated the unique ability of AASG to adapt class signatures to phenological differences by classifying a summer–winter image pair with a modest reduction in overall accuracy (κ=0.66). Observed sensitivity to c supported the hypothesis of an optimum value yielding enough training sites to describe class spectral variability, but conservative enough to minimize contamination of signatures due to classification errors. AASG offers significant advantages over traditional signature extension, particularly for temporally irregular time series. Although we demonstrated a simple implementation, the AASG approach is flexible and we outline several refinements which stand to improve performance. This development represents significant progress towards realizing the potential of long-term data archives to gain long-term understandings of global land cover dynamics.

Updating Land-Cover Maps by Classification of Image Time Series: A Novel Change-Detection-Driven Transfer Learning Approach

This paper proposes a novel change-detection-driven transfer learning (TL) approach to update land-cover maps by classifying remote-sensing images acquired on the same area at different times (i.e., image time series). The proposed approach requires that a reliable training set is available only for one of the images (i.e., the source domain) in the time series whereas it is not for another image to be classified (i.e., the target domain). Unlike other literature TL methods, no additional assumptions on either the similarity between class distributions or the presence of the same set of land-cover classes in the two domains are required. The proposed method aims at defining a reliable training set for the target domain, taking advantage of the already available knowledge on the source domain. This is done by applying an unsupervised-change-detection method to target and source domains and transferring class labels of detected unchanged training samples from the source to the target domain to initialize the target-domain training set. The training set is then optimized by a properly defined novel active learning (AL) procedure. At the early iterations of AL, priority in labeling is given to samples detected as being changed, whereas in the remaining ones, the most informative samples are selected from changed and unchanged unlabeled samples. Finally, the target image is classified. Experimental results show that transferring the class labels from the source domain to the target domain provides a reliable initial training set and that the priority rule for AL results in a fast convergence to the desired accuracy with respect to Standard AL.