Spatiotemporal Fusion Research Articles

A hybrid 3DSE-CNN-2DLSTM model for compound fault detection of wind turbines

In recent years, intelligent fault detection methods have achieved dramatic results for wind power generation. However, a majority of the available intelligent detected methods can only detect single faults. The supervisory control and data acquisition (SCADA) system is widely applied in wind turbines (WTs). SCADA data is a timestream of variable data. Considering spatial correlation and temporal correlation among SCADA data, this study proposes a WT compound fault detection model combining a three-dimensional squeeze-excitation convolutional neural network (3DSE-CNN), and a two-dimensional long and short-term memory network (2DLSTM). The proposed 3DSE-CNN-2DLSTM model consists of three parts: data preprocessing, spatiotemporal feature extraction, and fault detection. First, one-dimensional data is converted into a 2D image by a sliding window method. Then, in the proposed 3DSE-CNN model, the SE attention module is applied to enhance the convolutional channel information and extract spatial features. The proposed 2DLSTM model extracts spatiotemporal fusion features. Finally, the label of the fault category is obtained by argmax function. To validate 3DSE-CNN-2DLSTM, a 3 MW WT SCADA dataset from the south coast of Ireland is applied in the experiment. The proposed 3DSE-CNN-2DLSTM model achieves specific compound fault detection and improves detection accuracy.

Dynamic multi-fusion spatio-temporal graph neural network for multivariate time series forecasting

Multivariate time series data with multiple points is a kind of spatio-temporal data, and in recent years, spatio-temporal graph neural networks (STGNNs) have become one of the most effective methods for spatio-temporal data prediction. By leveraging graph neural networks and sequence models, STGNNs simultaneously capture the hidden temporal and spatial patterns in spatio-temporal data. However, there exist two key limitations: (1) Most existing STGNNs whether based on static graphs or dynamic graphs, fail to introduce time periodicity into graph structure learning and ignore the importance of multi-relational graph structure learning. (2) Most multivariate time series prediction models with multiple points ignore the relation between multiple attributes. To this end, we propose a novel framework called the Dynamic Multi-fusion Spatio-temporal Graph Neural Network (DMF-STNet), where multi-fusion embodies two meanings: spatio-temporal fusion and multi-attribute fusion, as well as multi-graph fusion and multi-layer fusion. The dynamic periodic graph is proposed to mine time periodicity that is hidden in graph structures, and we further offer multi-relational graph structure learning methods to capture static and dynamic relations. Moreover, we introduce a dual-stage feature aggregation method to aggregate hidden features from both primary and auxiliary attributes. Extensive experiments on three real-world datasets demonstrate the efficacy of our proposed DMF-STNet, and the visualization study proves the interpretability of learned weights.

Tourism demand forecasting: a deep learning model based on spatial-temporal transformer

Purpose Incorporating dynamic spatial effects exhibits considerable potential in improving the accuracy of forecasting tourism demands. This study aims to propose an innovative deep learning model for capturing dynamic spatial effects. Design/methodology/approach A novel deep learning model founded on the transformer architecture, called the spatiotemporal transformer network, is presented. This model has three components: the temporal transformer, spatial transformer and spatiotemporal fusion modules. The dynamic temporal dependencies of each attraction are extracted efficiently by the temporal transformer module. The dynamic spatial correlations between attractions are extracted efficiently by the spatial transformer module. The extracted dynamic temporal and spatial features are fused in a learnable manner in the spatiotemporal fusion module. Convolutional operations are implemented to generate the final forecasts. Findings The results indicate that the proposed model performs better in forecasting accuracy than some popular benchmark models, demonstrating its significant forecasting performance. Incorporating dynamic spatiotemporal features is an effective strategy for improving forecasting. It can provide an important reference to related studies. Practical implications The proposed model leverages high-frequency data to achieve accurate predictions at the micro level by incorporating dynamic spatial effects. Destination managers should fully consider the dynamic spatial effects of attractions when planning and marketing to promote tourism resources. Originality/value This study incorporates dynamic spatial effects into tourism demand forecasting models by using a transformer neural network. It advances the development of methodologies in related fields.

Exploring reliable infrared object tracking with spatio-temporal fusion transformer

Currently, most thermal infrared (TIR) trackers rely on feature matching between the search image and a fixed template cropped from the first frame. Some Siam-based TIR trackers with a template update mechanism introduce historical prediction information in the temporal dimension through correlation filters. However, their feature characterization capability is inadequate to resist target scale variations, appearance changes, and occlusion. To address this challenge, we explore a novel spatio-temporal fusion Transformer (STFT) model to realize robust TIR object tracking. Our approach involves a Transformer-based encoder–decoder that fuses spatio-temporal information. Specifically, we design a dynamic template update strategy based on salient points feature(SPF) representation, which allows the model to leverage the most powerful spatio-temporal information by retrieving multiple salient points on the target image. To further fortify the dynamic template update strategy, we propose an IoU-Aware target state estimation head that utilizes the joint representation of target classification and localization. An IoU-Aware criterion is developed for quality estimation of the dynamic template. The proposed STFT-Net approach has been put to the evaluation on three challenging benchmarks, with extensive experimental results showcasing its superior performance in contrast to acclaimed tracking algorithms. The code is available at https://github.com/qinxin-wh/STFT-Net.

A spatio-temporal fusion strategy for improving the estimation accuracy of the aboveground biomass in grassland based on GF-1 and MODIS

The accurate estimation of the aboveground biomass (AGB) is crucial for the sustainable utilization and management of grassland resources. Spatio-temporal inconsistencies between field samples and satellite images are major source of error in the estimation of grassland AGB. To solve this problem, this study selected the Three Rivers Headwater Region as the study area and proposed a selection strategy for base-period images of spatio-temporal fusion that was suitable for use at a large spatial scale in which cloud-free images are difficult to obtain. The spatial and temporal adaptive reflectance fusion model (STARFM) based on the selection strategy was used to generate a synthetic Normalized Difference Vegetation Index (NDVI) dataset with high spatial-temporal resolution by using the maximum value composite of GF-1 NDVI and MODIS NDVI to enhance the spatial-temporal quality of the images for field-scale application. Three estimation models for grassland AGB were then constructed by the random forest algorithm using synthetic NDVI, MODIS NDVI, GF-1 NDVI respectively, together with ancillary data. Following this, the estimation model with the highest accuracy was used to generate a 16-m eight-day time-series AGB in the growing season. The results showed: (1) The synthetic NDVI was correlated closely with the observed GF-1 NDVI, with an average R of 0.825 and a RMSE of 0.087. The temporal trend of the synthetic NDVI for each grassland type was highly consistent with that of the MODIS NDVI in the growing season with a correlation higher than 0.9. (2) The synthetic NDVI reduced the spatial difference between field samples and images to 16-fold, and the temporal difference was controlled to within four days under ideal conditions. (3) The synthetic NDVI improved the estimation accuracy of grassland AGB by about 15.9% and 19.7% (R2), and 13.7% and 17.5% (RMSE) relative to MODIS NDVI and GF-1 NDVI, respectively. (4) The time-series AGB revealed accurately the spatial distribution of and seasonal temporal variations in the grassland biomass. The results of this study may serve as scientific guidance for timely monitoring of grassland conditions and precise management of grassland resources in China.

Novel spatiotemporal graph attention model for production prediction and energy structure optimization of propylene production processes

Propylene is essential to the national economy. However, the propylene production process data have complex spatiotemporal, nonlinear and dynamic features, and the consideration of spatiotemporal effects can predict the propylene production more accurately. Therefore, a novel graph attention network (GAT) based long short-term memory network (LSTM)(GAT-LSTM) is proposed for the production prediction and energy structure optimization of propylene production processes. The GAT extracts spatial features to obtain correlations between variables, while the improved LSTM model extracts potential temporal-dependent features. The collected spatiotemporal interaction features are fused by the spatiotemporal fusion module and the adaptive control is realized. Finally, a fully connected layer maps spatiotemporal interaction features to the target domain for obtaining the output. Experimental results show that the GAT-LSTM outperforms the Conv-LSTM, the LSTM, the GAT, the Attribute-relevant distributed variational autoencoder and the Attention-LSTM. Additionally, the GAT-LSTM can provide the optimal raw material scheme for actual propylene production processes.

A Spatiotemporal Fusion Model of Land Surface Temperature Based on Pixel Long Time-Series Regression: Expanding Inputs for Efficient Generation of Robust Fused Results

Spatiotemporal fusion technology effectively improves the spatial and temporal resolution of remote sensing data by fusing data from different sources. Based on the strong time-series correlation of pixels at different scales (average Pearson correlation coefficients > 0.95), a new long time-series spatiotemporal fusion model (LOTSFM) is proposed for land surface temperature data. The model is distinguished by the following attributes: it employs an extended input framework to sidestep selection biases and enhance result stability while also integrating Julian Day for estimating sensor difference term variations at each pixel location. From 2013 to 2022, 79 pairs of Landsat8/9 and MODIS images were collected as extended inputs. Multiple rounds of cross-validation were conducted in Beijing, Shanghai, and Guangzhou with an all-round performance assessment (APA), and the average root-mean-square error (RMSE) was 1.60 °C, 2.16 °C and 1.71 °C, respectively, which proved the regional versatility of LOTSFM. The validity of the sensor difference estimation based on Julian days was verified, and the RMSE accuracy significantly improved (p < 0.05). The accuracy and time consumption of five different fusion models were compared, which proved that LOTSFM has stable accuracy performance and a fast fusion process. Therefore, LOTSFM can provide higher spatiotemporal resolution (30 m) land surface temperature research data for the evolution of urban thermal environments and has great application potential in monitoring anthropogenic heat pollution and extreme thermal phenomena.

STEPSBI: Quick spatiotemporal fusion with coarse- and fine-resolution scale transformation errors and pixel-based synthesis base image pair

Spatiotemporal fusion (STF) is crucial for reconciling the conflict between temporal and spatial resolutions of remote sensing observations. However, fusing images in heterogeneous areas remains challenging under continuous missing values. Moreover, most current STF methods only consider temporal errors and disregard the spatial scale error in time variation. Therefore, we proposed a Quick Spatiotemporal Fusion with Coarse- and Fine-Resolution Scale Transformation Errors and Pixel-Based Synthesis Base Image Pair (STEPSBI). First, the optimal pixel-based image synthesis strategy was designed using all available fine- and coarse-resolution images. Then, the scale transformation error (STE) of coarse-resolution downscaling to fine-resolution in temporal variation was quantified. And a residual term was introduced to reduce the prediction error from the temporal variation. Finally, the spatial scale and temporal errors were corrected using the results of super-pixel segmentation as spatial weights. This model has two strengths: (1) Pixel-based image synthesis alleviates the absence of base images under continuous missing values; and (2) STE correction restores spatial details of heterogeneous areas in rapid land cover change. In scenarios of continuous missing images, abrupt land cover changes, and high-resolution heterogeneity, we evaluated STEPSBI, the Gap Filling and Savitzky–Golay filtering method (GF-SG), the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), the Flexible Spatiotemporal DAta Fusion (FSDAF), the cross-attention-based adaptive weighting fusion network (CAFE), and the Multi-scene Spatiotemporal Fusion Network (MUSTFN). The results indicate that STEPSBI yields better overall performance than other models in cropland, woodland, grassland, and other land cover types. Furthermore, ablation experiments demonstrated that each component improved the model’s performance. In addition, STEPSBI had higher fusion efficiency because it was developed on the Google Earth Engine cloud computing platform. Therefore, STEPSBI was feasible for advancing fine monitoring of spatiotemporal image fusion under continuous missing values and heterogeneous surfaces. The program of STEPBSI is freely available at: https://code.earthengine.google.com/684844aa42f64fa6b4eebe3bc0dd6483.

Agri-Fuse: A novel spatiotemporal fusion method designed for agricultural scenarios with diverse phenological changes

Images of key phenological periods play a vital role in agricultural applications as they capture the unique spectral characteristics of crops. Unfortunately, acquiring high-spatial-resolution images of the key phenological period from a single satellite platform remains challenging due to its short duration and synchronization with the rainy season. Spatiotemporal fusion (STF) is an effective tool for the prediction of missing high spatial resolution images on the required date, however, most STF methods assume a uniform relationship between the reflectance on the base date and the predicted date for the same land-cover type, which fails to hold in agricultural scenarios because of the diverse phenological changes among different crop types or different growth processes, even within a single crop type. To address these challenges, we propose a novel spatiotemporal fusion method for agricultural scenarios called Agri-Fuse. Agri-Fuse emphasizes change information and classifies change types based on the difference image derived from the fine image on the base date and the coarse image on the predicted date. Considering the various regression relationships between reflectance on the base date and the predicted date for different change types, a category-based linear regression model was built, and we introduced an unmixing model to solve these regression coefficients. We validated the performance of Agri-Fuse at two experimental sites where diverse phenological changes occurred during the key phenological periods of the same and different crop types. Compared to the four typical fusion methods (FSDAF, RASDF, STARFM, and Fit-FC), Agri-Fuse achieved the best performance in preserving both spectral fidelity and spatial details. Ablation experiments further confirmed the effectiveness of the proposed change-type classification and category-based regression in capturing diverse phenological change information. Thus, Agri-Fuse is expected to precisely reconstruct high-spatial-resolution images of key phenological periods necessary for agricultural applications.

An Adaptive Multiscale Generative Adversarial Network for the Spatiotemporal Fusion of Landsat and MODIS Data

The monitoring of rapidly changing land surface processes requires remote sensing images with high spatiotemporal resolution. As remote sensing satellites have different satellite orbits, satellite orbital velocities, and sensors, it is challenging to acquire remote sensing images with high resolution and dense time series within a reasonable temporal interval. Remote sensing spatiotemporal fusion is one of the effective ways to acquire high-resolution images with long time series. Most of the existing STF methods use artificially specified fusion strategies, resulting in blurry images and poor generalization ability. Additionally, some methods lack continuous time change information, leading to poor performance in capturing sharp changes in land covers. In this paper, we propose an adaptive multiscale network for spatiotemporal fusion (AMS-STF) based on a generative adversarial network (GAN). AMS-STF reconstructs high-resolution images by leveraging the temporal and spatial features of the input data through multiple adaptive modules and multiscale features. In AMS-STF, for the first time, deformable convolution is used for the STF task to solve the shape adaptation problem, allowing for adaptive adjustment of the convolution kernel based on the different shapes and types of land use. Additionally, an adaptive attention module is introduced in the networks to enhance the ability to perceive temporal changes. We conducted experiments comparing AMS-STF to the most widely used and innovative models currently available on three Landsat-MODIS datasets, as well as ablation experiments to evaluate some innovative modules. The results demonstrate that the adaptive modules significantly improve the fusion effect of land covers and enhance the clarity of their boundaries, which proves the effectiveness of AMS-STF.

Spatio-temporal fusion and contrastive learning for urban flow prediction

Urban flow prediction is critical for urban planning, management, and safety. However, owing to the inherent instability of urban flows, prediction accuracy requires the fusion of multi-view influencing factors. Current prediction methods are insensitive to periodic changes in urban flows, and rarely consider the implied spatial and temporal correlations between similar functional areas. Thus, we propose a method based on spatiotemporal fusion and contrastive learning. We construct an extraction module of spatial and temporal views based on contrastive learning in the spatial and temporal dimensions. With the temporal view extraction, we can obtain the distribution and change in the global urban flow periodicity. Using the spatial view extraction, we can obtain the implied flow variation relationship between similar regions. Therefore, our model can capture the high-level semantic features of urban flow changes from multiple views. We also design a multi-view fusion and prediction network to combine multi-view representations that impact urban flow. We experimentally evaluated our proposed approach using two real-world datasets and demonstrated state-of-the-art forecasting performance, as well as effectiveness in resource-limited environments.

Fight detection with spatial and channel wise attention‐based ConvLSTM model

AbstractAn automated detection of aggressive and violent behaviour in videos has immense potential. It enables efficient online content filtering by restricting access to extreme content and also, when integrated with security systems, helps to monitor violence in surveillance videos. In this work, a convolutional neural network is combined with the proposed Spatial and Channel wise Attention‐based ConvLSTM encoder (SCan‐ConvLSTM). The proposed architecture performs an efficient spatiotemporal fusion of the features extracted from the video sequences containing fight scenes. In order to focus selectively on regions of utmost importance, this blended attention mechanism adjusts the weights of outputs in different locations and across different channels. This recurrent attention mechanism enhances the sequential refinement of activation maps and boosts the model performance. Finally, the experimental results have been presented that show the proposed architecture achieves superior results on the benchmark datasets (RWF‐2000, Violent‐flow, Hockey‐fights, and Movies).

Applying deep-learning enhanced fusion methods for improved NDVI reconstruction and long-term vegetation cover study: A case of the Danjiang River Basin

The Normalized Difference Vegetation Index (NDVI) is an essential metric in vegetation monitoring for remote sensing applications. While there are numerous long-term low-resolution NDVI datasets available there remains an unmet need for high-resolution NDVI reconstructions over extended time frames. Existing research has not comprehensively assessed the efficacy of various temporal fusion techniques for NDVI reconstruction at large regional scales. Traditional Spatiotemporal Image Fusion (TSTIF) methods often suffer from limited fusion accuracy due to input data quality constraints. To address these limitations this study introduces an innovative Deep Learning-Enhanced Spatiotemporal Fusion Method. Deep learning algorithms are employed to refine the spatial resolution of input data thereby facilitating the separation of complex image elements located at feature boundaries. This improved approach significantly enhances fusion accuracy as validated against five established TSTIF techniques through empirical analysis. As a case study we generate long-term Fractional Vegetation Cover (FVC) datasets to investigate the ecological dynamics of the Danjiang River Basin. Our findings reveal substantial gains in NDVI reconstruction accuracy through the incorporation of deep learning into traditional fusion techniques. Among the methods tested the STRUM algorithm showed the greatest improvement with its R2 value increasing from 0.872 to 0.894 and a consistent reduction in Root Mean Square Error (RMSE). The FSDAF technique emerged as the most effective boasting an R2 value of 0.953 and an RMSE of 0.012. These enhanced NDVI datasets provide more accurate representations of the basin's vegetation cover thus enriching long-term observational archives and facilitating further quantitative remote sensing analyses. Moreover we report a 21-year mean FVC of 0.702 for the Danjiang River Basin characterized by a “north-high south-low” distribution that varies seasonally. Improvements in vegetation cover were observed in Xichuan and Neixiang counties while environmental degradation is evident in the broader Danjiang River watershed. Key factors influencing these spatial variations include elevation and soil composition in addition to climate variables such as rainfall and temperature. Infrastructure projects like the South-to-North Water Diversion Project and targeted ecological conservation policies have also positively impacted vegetation within the basin.

Adaptability Evaluation of the Spatiotemporal Fusion Model in the Summer Maize Planting Area of the Southeast Loess Plateau

Precise regional crop yield estimates based on the high-spatiotemporal-resolution remote sensing data are essential for directing agronomic practices and policies to increase food security. This study used the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM), the flexible spatiotemporal data fusion (FSADF), and the spatial and temporal non-local filter based fusion model (STNLFFM) to calculate the normalized differential vegetation index (NDVI) of the summer maize planting area in the Southeast Loess Plateau based on the Sentinel-2 and MODIS data. The spatiotemporal resolution was 10 m and 1 d, respectively. Then, we evaluated the adaptability of the ESTARFM, FSADF, and STNLFFM fusion models in the field from the perspectives of spatial and textural characteristics of the data, summer maize NDVI growing curves, and yield estimation accuracy through qualitative visual discrimination and quantitative statistical analysis. The results showed that the fusion of ESTARFM–NDVI, FSDAF–NDVI, and STNLFFM–NDVI could precisely represent the variation tendency and local mutation information of NDVI during the growth period of summer maize, compared with MODIS–NDVI. The correlation between STNLFFM–NDVI and Sentinel-2–NDVI was favorable, with large correlation coefficients and a small root mean square error (RMSE). In the NDVI growing curve simulation of summer maize, STNLFFM introduced overall weights based on non-local mean filtering, which could significantly improve the poor fusion results at seedling and maturity stages caused by the long gap period of the high-resolution data in ESTARFM. Moreover, the accuracy of yield estimation was as follows (from high to low): STNLFFM (R = 0.742, mean absolute percentage error (MAPE) = 6.22%), ESTARFM (R = 0.703, MAPE = 6.80%), and FSDAF (R = 0.644, MAPE = 10.52%). The FADSF fusion model was affected by the spatial heterogeneity in the semi-humid areas, and the yield simulation accuracy was low. In the semi-arid areas, the FADSF fusion model had the advantages of less input data and a faster response.

Multiple Distilling-based spatial-temporal attention networks for unsupervised human action recognition

Unsupervised action recognition based on spatiotemporal fusion feature extraction has attracted much attention in recent years. However, existing methods still have several limitations: (1) The long-term dependence relationship is not effectively extracted at the time level. (2) The high-order motion relationship between non-adjacent nodes is not effectively captured at the spatial level. (3) The model complexity is too high when the cascade layer input sequence is long, or there are many key points. To solve these problems, a Multiple Distilling-based spatial-temporal attention (MD-STA) networks is proposed in this paper. This model can extract temporal and spatial features respectively and fuse them. Specifically, we first propose a Screening Self-attention (SSA) module; this module can find long-term dependencies in distant frames and high-order motion patterns between non-adjacent nodes in a single frame through a sparse metric on dot product pairs. Then, we propose the Frames and Keypoint-Distilling (FKD) module, which uses extraction operations to halve the input of the cascade layer to eliminate invalid key points and time frame features, thus reducing time and memory complexity. Finally, the Dim-reduction Fusion (DRF) module is proposed to reduce the dimension of existing features to further eliminate redundancy. Numerous experiments were conducted on three distinct datasets: NTU-60, NTU-120, and UWA3D, showing that MD-STA achieves state-of-the-art standards in skeleton-based unsupervised action recognition.

Spatiotemporal Fusion Model of Remote Sensing Images Combining Single-Band and Multi-Band Prediction

In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels’ convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies.

STF-Trans: A two-stream spatiotemporal fusion transformer for very high resolution satellites images

Spatiotemporal satellite image fusion is regarded as an effective approach to address the limitations of a single optical sensor and generate data with high spatial and temporal resolutions. Thanks to the recent advances of Deep Learning, spatiotemporal fusion has gained a significant performance improvement. However, most models require at least three inputs, including a high resolution one at a prior date, to make the prediction at the desired date. Such a requirement is not always guaranteed in practice, in particular due to the high cost of high resolution image and bad atmospheric conditions. In the last few years, Transformers have achieved tremendous performance on several computer vision tasks compared to convolutional neural networks. Inspired by the new trend, we proposed an end-to-end two-stream SpatioTemporal fusion technique based on encoder–decoder transformer architecture, termed STF-Trans. To deal with a complexity and high-cost of data preparation, the proposed approach aims to fuse a coarse-resolution image with high temporal resolution, and a very high resolution Google Earth image, to produce a very high resolution image at the desired date. The proposed technique employs a two-stream convolutional network to capture both temporal and spatial features from the inputs, and then the encoder–decoder transformer captures long-range dependency of the latter and generates a more efficient image representation from the shallow features. Finally, a shallow network maps the features into the desired fused image. The experimental results show that the proposed method outperforms the current state-of-the-art techniques at both quantitative and qualitative evaluations.

Improving agricultural field parcel delineation with a dual branch spatiotemporal fusion network by integrating multimodal satellite data

Accurate spatial information for agricultural field parcels is important for agricultural production management and understanding agro-industrialization and intensification. However, traditional remote sensing methods that rely on single-modal or single-date data struggle to identify heterogeneous field parcels, particularly in regions dominated by smallholder farming systems. To address this challenge, we proposed a Dual branch Spatiotemporal Fusion Network (DSTFNet) that integrated very high-resolution (VHR) images and medium-resolution satellite image time series (MRSITS) to extract agricultural field parcels over various landscapes. The DSTFNet consisted of two branches: a spatial branch that extracted spatial features from VHR images and a temporal branch that explored seasonal spectral dynamics from MRSITS data by using ConvLSTM units and an attention module. We evaluated the DSTFNet in four regions across China by using GF-2 and Sentinel-2 data. The results showed that DSTFNet performed well in delineating agricultural field parcels, achieving the highest Matthew’s correlation coefficient (MCC) = 0.823 for the field extent, the highest F1-score of edge (Fedge) = 0.865 for field boundary, and the lowest errors of segmentation evaluation index (SEI) = 0.191 for the vectorized field parcels in Hubei province. In addition, DSTFNet significantly outperformed two single-branch models that used VHR or MRSITS alone, as well as existing BsiNet, ResUNet_a, UNet and RAUNet models. DSTFNet also showed good spatial transferability in distinct regions without training data (on average, MCC = 0.728, Fedge = 0.729, and SEI = 0.281 for three target regions). Using limited training data to fine-tune the DSTFNet can further improve its ability to delineate field parcels over complex regions. The visualization analysis of temporal attention weights demonstrated that DSTFNet can well capture cropland spectral dynamics, making it advantageous in extracting diverse cropland parcels. By exploiting important spectral, spatial and temporal information from multimodal satellite data, DSTFNet provided an effective, robust, and transferable solution for accurately delineating agricultural field parcels across heterogeneous farming systems.

Spatio-temporal fusion of meteorological factors for multi-site PM2.5 prediction: A deep learning and time-variant graph approach

In the field of environmental science, traditional methods for predicting PM2.5 concentrations primarily focus on singular temporal or spatial dimensions. This approach presents certain limitations when it comes to deeply mining the joint influence of multiple monitoring sites and their inherent connections with meteorological factors. To address this issue, we introduce an innovative deep-learning-based multi-graph model using Beijing as the study case. This model consists of two key modules: firstly, the 'Meteorological Factor Spatio-Temporal Feature Extraction Module'. This module deeply integrates spatio-temporal features of hourly meteorological data by employing Graph Convolutional Networks (GCN) and Long Short-Term Memory (LSTM) for spatial and temporal encoding respectively. Subsequently, through an attention mechanism, it retrieves a feature tensor associated with air pollutants. Secondly, these features are amalgamated with PM2.5 concentration values, allowing the 'PM2.5 Concentration Prediction Module' to predict with enhanced accuracy the joint influence across multiple monitoring sites. Our model exhibits significant advantages over traditional methods in processing the joint impact of multiple sites and their associated meteorological factors. By providing new perspectives and tools for the in-depth understanding of urban air pollutant distribution and optimization of air quality management, this model propels us towards a more comprehensive approach in tackling air pollution issues.

Construction of customer-aware grid interactive service system based on neural network model

Abstract Neural network models have advantages in spatial and temporal information processing, which provide new ideas for the construction of customer-aware grid interactive service system. In this paper, firstly, the spatio-temporal fusion deep neural network model based on the attention mechanism is used to extract spatial and temporal information by convolutional neural network and gated recurrent neural network, respectively. The attention mechanism is also introduced to assign weights to various types of features to improve prediction accuracy. Secondly, the global information of the attention mechanism of the channel on the feature map is fully utilized to solve the problem of inadequate extraction of feature information from a single model, and then the grid interactive service system is constructed. The results show that the model proposed in this paper has higher prediction accuracy with root mean square error, coefficient of determination, and prediction accuracy of 0.972, 1.742, and 0.935, respectively. To verify the performance of the cache-based curve service in handling customer-perceived grid interactions, minute samples are created, where the maximum number of minute sample records is 3.5 million. Thus, it is shown that by introducing the attention mechanism can effectively improve the computational efficiency of the model, get better prediction results, and meet the demand for efficient and stable customer-aware interactive service system.