Spatiotemporal Fusion Method Research Articles

“Regression-then-Fusion” or “Fusion-then-Regression”? A Theoretical Analysis for Generating High Spatiotemporal Resolution Land Surface Temperatures

The trade-off between spatial and temporal resolutions in satellite sensors has inspired the development of numerous thermal sharpening methods. Specifically, regression and spatiotemporal fusion are the two main strategies used to generate high-resolution land surface temperatures (LSTs). The regression method statically downscales coarse-resolution LSTs, whereas the spatiotemporal fusion method can dynamically downscale LSTs; however, the resolution of downscaled LSTs is limited by the availability of the fine-resolution LSTs. Few studies have combined these two methods to generate high spatiotemporal resolution LSTs. This study proposes two strategies for combining regression and fusion methods to generate high spatiotemporal resolution LSTs, namely, the “regression-then-fusion” (R-F) and “fusion-then-regression” (F-R) methods, and discusses the criteria used to determine which strategy is better. The R-F and F-R have several advantages: (1) they fully exploit the information in the available data on the visible and near infrared (VNIR) and thermal infrared (TIR) bands; (2) they downscale the LST time series to a finer resolution corresponding to that of VNIR data; and (3) they inherit high spatial reconstructions from the regression method and dynamic temporal reconveyance from the fusion method. The R-F and F-R were tested with different start times and target times using Landsat 8 and Advanced Spaceborne Thermal Emission and Reflection Radiometer data. The results showed that the R-F performed better than the F-R when the regression error at the start time was smaller than that at the target time, and vice versa.

An Enhanced Single-Pair Learning-Based Reflectance Fusion Algorithm with Spatiotemporally Extended Training Samples

Spatiotemporal fusion methods are considered a useful tool for generating multi-temporal reflectance data with limited high-resolution images and necessary low-resolution images. In particular, the superiority of sparse representation-based spatiotemporal reflectance fusion model (SPSTFM) in capturing phenology and type changes of land covers has been preliminarily demonstrated. Meanwhile, the dictionary training process, which is a key step in the sparse learning-based fusion algorithm, and its effect on fusion quality are still unclear. In this paper, an enhanced spatiotemporal fusion scheme based on the single-pair SPSTFM algorithm has been proposed through improving the process of dictionary learning, and then evaluated using two actual datasets, with one representing a rural area with phenology changes and the other representing an urban area with land cover type changes. The validated strategy for enhancing the dictionary learning process is divided into two modes to enlarge the training datasets with spatially and temporally extended samples. Compared to the original learning-based algorithm and other employed typical single-pair-based fusion models, experimental results from the proposed fusion method with two extension modes show improved performance in modeling reflectance using the two preceding datasets. Furthermore, the strategy with temporally extended training samples is more effective than the strategy with spatially extended training samples for the land cover area with phenology changes, whereas it is opposite for the land cover area with type changes.

An Enhanced Linear Spatio-Temporal Fusion Method for Blending Landsat and MODIS Data to Synthesize Landsat-Like Imagery

Landsat and MODIS data have been widely utilized in many remote sensing applications, however, the trade-off between the spatial resolution and temporal frequency has limited their capacities in monitoring detailed spatio-temporal dynamics. Spatio-temporal fusion methods based on a linear model that considers the differences between fine- and coarse-spatial-resolution images as linear can effectively solve this trade-off problem, yet the existing linear fusion methods either regard the coefficients of the linear model as constants or have adopted regression methods to calculate the coefficients, both of which may introduce some errors in the fusion process. In this paper, we proposed an enhanced linear spatio-temporal fusion method (ELSTFM) to improve the data fusion accuracy. In the ELSTFM, it is not necessary to calculate the slope of the linear model, and the intercept, which can be deemed as the residual caused by systematic biases, is calculated based on spectral unmixing theory. Additionally, spectrally similar pixels in a given fine-spatial-resolution pixel’s neighborhood and their corresponding weights were used in the proposed method to mitigate block effects. Landsat-7/ETM+ and 8-day composite MODIS reflectance data covering two study sites with heterogeneous and homogenous landscapes were selected to validate the proposed method. Compared to three other typical spatio-temporal fusion methods visually and quantitatively, the predicted images obtained from ELSTFM could acquire better results for the two selected study sites. Furthermore, the resampling methods used to resample MODIS to the same spatial resolution of Landsat could slightly, but did not significantly influence the fusion accuracy, and the distributions of slopes of different bands for the two study sites could all be deemed as normal distributions with a mean value close to 1. The performance of ELSTFM depends on the accuracy of residual calculation at fine-resolution and large landscape changes may influence the fusion accuracy.

Spatiotemporal Satellite Image Fusion Using Deep Convolutional Neural Networks

We propose a novel spatiotemporal fusion method based on deep convolutional neural networks (CNNs) under the application background of massive remote sensing data. In the training stage, we build two five-layer CNNs to deal with the problems of complicated correspondence and large spatial resolution gaps between MODIS and Landsat images. Specifically, we first learn a nonlinear mapping CNN between MODIS and low-spatial-resolution (LSR) Landsat images and then learn a super-resolution CNN between LSR Landsat and original Landsat images. In the prediction stage, instead of directly taking the outputs of CNNs as the fusion result, we design a fusion model consisting of high-pass modulation and a weighting strategy to make full use of the information in prior images. Specifically, we first map the input MODIS images to transitional images via the learned nonlinear mapping CNN and further improve the transitional images to LSR Landsat images via the fusion model; then, via the learned SR CNN, the LSR Landsat images are supersolved to transitional images, which are further improved to Landsat images via the fusion model. Compared with the previous learning-based fusion methods, mainly referring to the sparse-representation-based methods, our CNNs-based spatiotemporal method has the following advantages: 1) automatically extracting effective image features; 2) learning an end-to-end mapping between MODIS and LSR Landsat images; and 3) generating more favorable fusion results. To examine the performance of the proposed fusion method, we conduct experiments on two representative Landsat-MODIS datasets by comparing with the sparse-representation-based spatiotemporal fusion model. The quantitative evaluations on all possible prediction dates and the comparison of fusion results on one key date in both visual effect and quantitative evaluations demonstrate that the proposed method can generate more accurate fusion results.

Spatiotemporal Fusion of MODIS and Landsat-7 Reflectance Images via Compressed Sensing

The fusion of remote sensing images with different spatial and temporal resolutions is needed for diverse Earth observation applications. A small number of spatiotemporal fusion methods that use sparse representation appear to be more promising than weighted- and unmixing-based methods in reflecting abruptly changing terrestrial content. However, none of the existing dictionary-based fusion methods consider the downsampling process explicitly, which is the degradation and sparse observation from high-resolution images to the corresponding low-resolution images. In this paper, the downsampling process is described explicitly under the framework of compressed sensing for reconstruction. With the coupled dictionary to constrain the similarity of sparse coefficients, a new dictionary-based spatiotemporal fusion method is built and named compressed sensing for spatiotemporal fusion, for the spatiotemporal fusion of remote sensing images. To deal with images with a high-resolution difference, typically Landsat-7 and Moderate Resolution Imaging Spectrometer (MODIS), the proposed model is performed twice to shorten the gap between the small block size and the large resolution rate. In the experimental procedure, the near-infrared, red, and green bands of Landsat-7 and MODIS are fused with root mean square errors to check the prediction accuracy. It can be concluded from the experiment that the proposed methods can produce higher quality than five state-of-the-art methods, which prove the feasibility of incorporating the downsampling process in the spatiotemporal model under the framework of compressed sensing.

Spatiotemporal Fusion of Remote Sensing Images with Structural Sparsity and Semi-Coupled Dictionary Learning

Fusion of remote sensing images with different spatial and temporal resolutions is highly needed by diverse earth observation applications. A small number of spatiotemporal fusion methods using sparse representation appear to be more promising than traditional linear mixture methods in reflecting abruptly changing terrestrial content. However, one of the main difficulties is that the results of sparse representation have reduced expressional accuracy; this is due in part to insufficient prior knowledge. For remote sensing images, the cluster and joint structural sparsity of the sparse coefficients could be employed as a priori knowledge. In this paper, a new optimization model is constructed with the semi-coupled dictionary learning and structural sparsity to predict the unknown high-resolution image from known images. Specifically, the intra-block correlation and cluster-structured sparsity are considered for single-channel reconstruction, and the inter-band similarity of joint-structured sparsity is considered for multichannel reconstruction, and both are implemented with block sparse Bayesian learning. The detailed optimization steps are given iteratively. In the experimental procedure, the red, green, and near-infrared bands of Landsat-7 and Moderate Resolution Imaging Spectrometer (MODIS) satellites are put to fusion with root mean square errors to check the prediction accuracy. It can be concluded from the experiment that the proposed methods can produce higher quality than state-of-the-art methods.

Regional-scale winter wheat phenology monitoring using multisensor spatio-temporal fusion in a South Central China growing area

Monitoring crop phenology is essential for evaluating crop productivity and crop management. Remote sensing provides an efficient way to monitor crop phenological metrics at a large-scale. However, the widely used AVHRR and MODIS images are less reliable at a small-scale and in areas with heterogeneous land covers, such as the patchy cropland in South Central China. Therefore, this study analyzed the spatial and temporal variations of winter wheat phenology in South Central China, using enhanced vegetation index (EVI) time series predicted by a spatio-temporal fusion method that combines information from Landsat and MODIS images. The 13-year predicted EVI showed a close correspondence with the EVI derived from original Landsat images. Start of season (SOS), peak greenness, and end of season (EOS) were derived from the predicted EVI time series. The comparison with ground observations showed that the differences between the predicted phenological metrics and observations were usually within seven days. The length of the growing season demonstrated high spatial heterogeneity over the study area and the spatial patterns varied from year to year. The phenological dates did not show obvious increasing or decreasing trends through 13 years. The length of the growing season in the study area was positively correlated with precipitation, but the duration from SOS to peak greenness and the duration from peak greenness to EOS were strongly and negatively correlated with hours of sunshine.

Fast and Accurate Spatiotemporal Fusion Based Upon Extreme Learning Machine

Spatiotemporal fusion is important in providing high spatial resolution earth observations with a dense time series, and recently, learning-based fusion methods have been attracting broad interest. These algorithms project image patches onto a feature space with the enforcement of a simple mapping to predict the fine resolution patches from the corresponding coarse ones. However, the sophisticated projection, e.g., sparse representation, is always computationally complex and difficult to be implemented on large patches, which cannot grasp enough local structural information in the coarse patches. To address these issues, a novel spatiotemporal fusion method is proposed in this letter, using a powerful learning technique, i.e., extreme learning machine (ELM). Unlike traditional approaches, we devote to learning a mapping function on difference images directly, rather than the sophisticated feature representation followed by a simple mapping. Characterized by good generalization performance and fast speed, the ELM is employed to achieve accurate and fast fine patches prediction. The proposed algorithm is evaluated by five actual data sets of Landsat enhanced thematic mapper plus-moderate resolution imaging spectroradiometer acquisitions and experimental results show that our method obtains better fusion results while achieving much greater speed.

Action recognition with spatio-temporal augmented descriptor and fusion method

Action recognition is one of the most popular fields of computer vision, and lots of efforts have been made to improve recognition accuracy. While multiple descriptors are extracted to represent action, the spatio-temporal information is lost. In order to incorporate spatio-temporal information, we propose a novel method called augmented descriptor by adding the information to the original descriptor. As descriptors represent different video features, such as static appearance and motion information, previous methods just concatenate various descriptors. However, we propose a fusion method to boost the recognition accuracy of action recognition. The Multiple Kernel Learning is utilized to fuse different descriptors to get better representation in our fusion method. We also evaluate the contribution of normalization method to recognition accuracy. Our proposed methods are tested on the benchmark datasets, Olympic Sports dataset and HMDB51 dataset. The experimental results show that our approaches outperform the baseline method of improved trajectories and are effective in recognizing various actions.

Fast spatiotemporal data fusion: merging LISS III with AWiFS sensor data

The high resolution of remote sensors has evolved to capture the fine details of the Earth’s surface features in remote-sensing (RS) data. There is a trade-off between this fine spatial resolution and the temporal resolution of global space-borne sensors. Global space-borne sensors are not good enough to acquire an image at fine spatial and high temporal (FSHT) resolutions simultaneously. In this article, we propose a computationally efficient technique to create a FSHT resolution image using a ground-based data processing system. Resourcesat-2, part of the Indian Space Research Organization’s (ISRO’s) mission, carries linear imaging self-scanners (LISS III and LISS IV) and an advanced wide-field sensor (AWiFS) on board. The spatial and temporal resolutions of LISS III are 23.5 m and 24 days and those of AWiFS are 56 m and 5 days, respectively. The proposed method creates a synthetic FSHT resolution image with 23.5 m spatial and 5 day temporal resolution. This method is referred to as ‘LISS III spatial and AWiFS temporal’ (LSAT) data fusion. The LSAT data fusion method is based on a sub-pixel relationship between the images of a single AWiFS–LISS III image pair, which was acquired before or after the ‘prediction date’: a synthetic LISS III image for the time is ‘predicted’ (synthesized) using an AWiFS image at time and a single AWiFS–LISS III image pair at time , where . The LSAT model was tested on simulated and real data sets acquired by the LISS III and AWiFS sensors. The proposed method was compared with the recently developed spatiotemporal data fusion methods. The experimental results demonstrate that the method is computationally efficient and shows consistent prediction accuracy in retrieving surface reflectance changes.

Land-surface temperature retrieval at high spatial and temporal resolutions based on multi-sensor fusion

Land-surface temperature (LST) is of great significance for the estimation of radiation and energy budgets associated with land-surface processes. However, the available satellite LST products have either low spatial resolution or low temporal resolution, which constrains their potential applications. This paper proposes a spatiotemporal fusion method for retrieving LST at high spatial and temporal resolutions. One important characteristic of the proposed method is the consideration of the sensor observation differences between different land-cover types. The other main contribution is that the spatial correlations between different pixels are effectively considered by the use of a variation-based model. The method was tested and assessed quantitatively using the different sensors of Landsat TM/ETM+, moderate resolution imaging spectroradiometer and the geostationary operational environmental satellite imager. The validation results indicate that the proposed multisensor fusion method is accurate to about 2.5 K.