Spatial Temporal Fusion Research Articles

STAFGCN:A spatial-temporal attention-based fusion graph convolution network for pedestrian trajectory prediction

Pedestrian trajectory prediction provides crucial data support for the development of smart cities. Existing pedestrian trajectory prediction methods often overlook the different types of pedestrian interactions and the micro-level spatial-temporal relationships when handling the interaction information in spatial dimension and temporal dimension. The model employs a spatial-temporal attention-based fusion graph convolutional framework to predict future pedestrian trajectories. For the different types of local and global relationships between pedestrians, it first employs spatial-temporal attention mechanisms to capture dependencies in pedestrian sequence data, obtaining the social interactions of pedestrians in spatial contexts and the movement trends of pedestrians over time. Subsequently, a fusion graph convolutional module merges the temporal weight matrix and the spatial weight matrix into a spatial-temporal fusion feature map. Finally, a decoder section utilizes TimeStacked Convolutional Neural Networks to predict future trajectories. The final validation on the ETH and UCY datasets yielded experimental results with an Average Displacement Error(ADE) of 0.34 and an Final Displacement Error(FDE) of 0.55. The visualization results further demonstrated the rationality of the model.

Integrating spatial and temporal features for enhanced artifact removal in multi-channel EEG recordings

Objective.Various artifacts in electroencephalography (EEG) are a big hurdle to prevent brain-computer interfaces from real-life usage. Recently, deep learning-based EEG denoising methods have shown excellent performance. However, existing deep network designs inadequately leverage inter-channel relationships in processing multi-channel EEG signals. Typically, most methods process multi-channel signals in a channel-by-channel way. Considering the correlations among EEG channels during the same brain activity, this paper proposes utilizing channel relationships to enhance denoising performance.Approach.We explicitly model the inter-channel relationships using the self-attention mechanism, hypothesizing that these correlations can support and improve the denoising process. Specifically, we introduce a novel denoising network, named spatial-temporal fusion network (STFNet), which integrates stacked multi-dimension feature extractor to explicitly capture both temporal dependencies and spatial relationships.Main results.The proposed network exhibits superior denoising performance, with a 24.27% reduction in relative root mean squared error compared to other methods on a public benchmark. STFNet proves effective in cross-dataset denoising and downstream classification tasks, improving accuracy by 1.40%, while also offering fast processing on CPU.Significance.The experimental results demonstrate the importance of integrating spatial and temporal characteristics. The computational efficiency of STFNet makes it suitable for real-time applications and a potential tool for deployment in realistic environments.

Driver fatigue detection method based on temporal–spatial adaptive networks and adaptive temporal fusion module

The reduction of traffic accidents by determining the driver’s state through fatigue detection is a worthy research issue. Most of the current fatigue driving detection method fails to fully utilize the temporal features of fatigue. To address this problem, this paper proposes a driver fatigue detection method combined with temporal–spatial adaptive networks (TSNet) and adaptive temporal fusion module (ATFM). First, a frame sequence of T frames is obtained by strided sampling of the input video and data enhancement. Subsequently, the temporal–spatial adaptive module (TSAM) is used as the core module and incorporated into Efficientnet-v2 to construct TSNet, which adaptively extracts temporal features according to different videos, adds attention weights to discriminative spatial and channel features, and fully extracts fatiguing temporal–spatial features of videos. Finally, ATFM is utilized to learn the weights between the fatigue classification scores of each frame in the frame sequence and adaptively fuses the fatigue classification scores of individual frames to obtain fatigue prediction results, increasing the extent of the influence of keyframes on the fatigue prediction results. In this paper, the proposed method achieves an accuracy of 89.42% on the NTHU-DDD dataset, which is better than other state-of-the-art methods, and the number of parameters of the proposed method is 24.70M, which is smaller than most of the methods. Through a series of comparative experiments, TSNet and ATFM alone also outperform models and modules with similar functionality.

Multi-layer adaptive spatial-temporal feature fusion network for efficient food image recognition

Numerous deep learning methods have been developed to tackle the challenges of recognizing food images, including convolutional neural networks, deep feature extraction, and deep feature fusion methods. This research proposes a new architecture called ASTFF-Net that uses deep feature fusion to tackle various challenges in food recognition, including similarity patterns between two categories, multi-object problems, light conditions, camera position, noise objects, and blurred images. ASTFF-Net is a robust and adaptive spatial–temporal fusion network designed to address these challenges effectively. The ASTFF-Net architecture consisted of three networks. In the spatial feature extraction network, the ResNet50 architecture was used to extract robust spatial features, and the reduction operation was utilized to minimize parameter size. Subsequently, the spatial features were passed through a 1D convolution (Conv1D) to fit the features into the recurrent neural networks. In the temporal feature extraction network, the spatial features were given to the long short-term memory, allowing the network to learn from various long sequence patterns. In the adaptive feature fusion network, the robust spatial and temporal features were fused and assigned to the Conv1D, followed by the softmax function. The ASTFF-Net architecture is also intended to decrease the number of network parameters and prevent overfitting problems. Experimental results on four benchmark food image datasets: Food11, UEC Food-100, UEC Food-256, and ETH Food-101, demonstrate that the proposed ASTFF-Nets, particularly ASTFF-NetB3, were more competitive compared with other existing methods.

Assessment of Ecosystem Service Values of Urban Wetland: Taking East Lake Scenic Area in Wuhan as an Example

Urban wetlands represent a significant ecosystem type within urban landscapes. The quantitative assessment of their ecological service value holds great significance in guiding and improving the urban habitat. However, due to the insufficient spatial resolution of traditional low-to-medium resolution remote sensing imagery for surface monitoring, previous studies have conducted relatively limited research on the ecosystem services of urban wetlands. In this paper, based on multi-source data including multi-scale remote sensing data, a spatial-temporal fusion model and multiple ecological parameter inversion models were employed to invert three key ecological parameters at high spatial resolution, thereby assessing the ecosystem service values (ESVs) of urban wetlands. Taking the East Lake Scenic Area (ELSA) in Wuhan as an example, the dynamics of its ecosystem services’ value components were comparatively analyzed. The results indicate that, while the total value of ecosystem services declined slightly in 2015 compared to 2011, there was a notable increase in their value to CNY 3.219 billion by 2019, which represents a doubling of the total value relative to 2011. This trend could be primarily attributed to a significant rise in cultural services within the region. Specifically, the value of tourism services reached CNY 2.090 billion in 2019, representing a threefold increase compared to 2011. This demonstrates that ecosystem services in the ELSA have been significantly optimized and enhanced through associated ecological projects. Further research should investigate the mechanisms by which urbanization affects these crucial ecosystem services, particularly the characterization of cultural services in urban wetlands, and develop more effective strategies to enhance urban resilience and sustainable development.

Spatial Decomposition and Temporal Fusion Based Inter Prediction for Learned Video Compression

Spatial Decomposition and Temporal Fusion Based Inter Prediction for Learned Video Compression

STFGCN: Spatial–temporal fusion graph convolutional network for traffic prediction

Accurate traffic prediction plays a crucial role in improving traffic conditions and optimizing road utilization. Effectively capturing the multi-scale temporal dependencies and dynamic spatial dependencies is crucial for accurate traffic prediction. These features can effectively reflect complex dynamic spatial–temporal processes, which have not been comprehensively addressed in most existing research work. Motivated by this issue, the primary contribution of this paper lies in proposing a novel Spatial–Temporal Fusion Graph Neural Network (STFGCN) for accurate traffic prediction, achieved by extracting multi-scale temporal dependencies from multiple semantic environments and constructing a dynamic adaptive graph to model spatial dependencies based on temporal characteristics. Specifically, to capture the multi-scale dynamic temporal dependencies effectively, a Multi-Scale Fusion Convolution (MSFC) module is designed, in which the temporal dependencies are extracted from multiple textual environments by utilizing multi-scale convolution. In order to model dynamic spatial dependencies, a Spatial Adaptive Fusion Convolution (SAFC) module is designed by combining the recent coherence and periodicity to infer dynamic graphs, which are then fused to model dynamic spatial dependencies. Extensive experimental results on five real-world datasets demonstrate that the proposed STFGCN has superior performance. Specifically, compared with the state-of-the-art baselines, STFGCN reduced 1.2% to 16.4% in RMSE measure.

A multi-feature spatial–temporal fusion network for traffic flow prediction

The traffic flow prediction is the key to alleviate traffic congestion, yet very challenging due to the complex influence factors. Currently, the most of deep learning models are designed to dig out the intricate dependency in continuous standardized sequences, which are dependent to high requirements for data continuity and regularized distribution. However, the data discontinuity and irregular distribution are inevitable in the real-world practical application, then we need find a way to utilize the powerful effect of the multi-feature fusion rather than continuous relation in standardized sequences. To this end, we conduct the prediction based on the multiple traffic features reflecting the complex influence factors. Firstly, we propose the ATFEM, an adaptive traffic features extraction mechanism, which can select important influence factors to construct joint temporal features matrix and global spatial features matrix according to the traffic condition. In this way, the feature’s representation ability can be improved. Secondly, we propose the MFSTN, a multi-feature spatial–temporal fusion network, which include the temporal transformer encoder and graph attention network to obtain the latent representation of spatial–temporal features. Especially, we design the scaled spatial–temporal fusion module, which can automatically learn optimal fusion weights, further adapt to inconsistent spatial–temporal dimensions. Finally, the multi-layer perceptron gets the mapping function between these comprehensive features and traffic flow. This method helps to improve the interpretability of the prediction. Experimental results show that the proposed model outperforms a variety of baselines, and it can accurately predict the traffic flow when the data missing rate is high.

DFSTrack: Dual-stream fusion Siamese network for human pose tracking in videos

Human pose tracking is a challenging task that involves estimating the human pose and tracking it across multiple frames in a video sequence. In recent years, deep learning-based methods have made significant progress in this field, achieving state-of-the-art performance. However, due to complex background and occlusion among people missed detection and incorrect association matching are still the challenging problems. To address these issues, we adopt a top-down framework to perform human pose tracking in the paper. We propose a human detection prediction recovery module (HDP module) to recover missed detection, and propose a dual-stream fusion Siamese network for human matching (DFSTrack). Specifically, we design a residual graph convolutional block (RGCN block) for spatial position encoding of human keypoints, and use spatial self-attention and temporal cross-attention to design a dual-stream spatial–temporal fusion transformer (DST Transformer). The graph convolutional block and transformer are cascaded to simultaneously obtain information on the spatial and temporal positions of human keypoints, allowing the Siamese network to solve the erroneous human matching. Experimental results on the PoseTrack17 dataset, PoseTrack18 dataset and PoseTrack21 dataset demonstrate that our proposed method outperforms state-of-the-art methods on human pose tracking tasks. Our code and pretrained models are available at https://github.com/yhtian2023/DFSTrack.

Spatial-temporal graph Transformer for object tracking against noise spoofing interference

Achieving accurate object tracking against noise spoofing interference caused by similar backgrounds, similar objects, occlusion, or illumination variations is a challenge, especially since Siamese trackers largely ignore the spatial-temporal and part-level information of object structure. To handle this problem, we present a novel object tracking network with spatial-temporal awareness, which learns the correlation of local feature changes and extracts global spatial-temporal fusion features of the object to prevent noise spoofing interference. Specifically, our tracker integrates an object graph reconstruction representation module and a spatial-temporal graph Transformer module. The graph reconstruction representation module models the object structure and search region as part-to-part graph node correspondences, propagating object information to achieve part-level feature aggregation in the search region. The proposed spatial-temporal graph Transformer module fuses the temporal and spatial features of the object to enhance the contextual features and correlation between temporal and spatial feature variations, recognizing the noise spoofing interference to enhance the tracking performance. Meanwhile, our tracker learns object motion information to improve object state awareness, enhancing tracking accuracy under similar objects interference. Extensive experiments on six public datasets validate that our tracker outperforms related state-of-the-art trackers and achieves accurate tracking results against noise spoofing interference.

Generating high-resolution hyperspectral time series datasets based on unsupervised spatial-temporal-spectral fusion network incorporating a deep prior

Over the past decade, image fusion has emerged as an indispensable tool for surface monitoring due to its capability to reconstruct high-quality surface reflectance. While satisfactory progress has been made in the spatial-spectral and spatial-temporal fusion of remote sensing images, the fusion of spatial-temporal-spectral data remains a challenging task. This challenge arises from the presence of multiple nonlinear relationships caused by inconsistent temporal, spatial, and spectral resolutions. In this paper, we propose a novel approach to address the spatial-temporal-spectral fusion of remote sensing images using a triple discriminator Generative Adversarial Network called TDGAN. Our approach represents the first attempt at spatial-temporal-spectral fusion based on deep learning techniques. Specifically, TDGAN achieves unsupervised learning by incorporating a deep spectral transformation prior, explores the application performance of the Swin Transformer in spectral super-resolution for the first time. Additionally, three discriminators are designed based on the prior consistencies: phenological consistency for data acquired at different times and spatial-spectral consistency for data acquired at the same time. These discriminators effectively enhance the prediction accuracy of missing hyperspectral images. Extensive simulations and real experiments were conducted to evaluate the effectiveness of our proposed TDGAN. The results demonstrate its robustness and comparable performance to other state-of-the-art methods. The proposed approach provides high-quality hyperspectral datasets with high spatial and temporal resolutions, facilitating continuous land cover observations.

STFEformer: Spatial–Temporal Fusion Embedding Transformer for Traffic Flow Prediction

In the realm of Intelligent Transportation Systems (ITSs), traffic flow prediction is crucial for multiple applications. The primary challenge in traffic flow prediction lies in the handling and modeling of the intricate spatial–temporal correlations inherent in transport data. In recent years, many studies have focused on developing various Spatial–Temporal Graph Neural Networks (STGNNs), and researchers have also begun to explore the application of transformers to capture spatial–temporal correlations in traffic data. However, GNN-based methods mainly focus on modeling spatial correlations statically, which significantly limits their capacity to discover dynamic and long-range spatial patterns. Transformer-based methods have not sufficiently extracted the comprehensive representation of traffic data features. To explore dynamic spatial dependencies and comprehensively characterize traffic data, the Spatial–Temporal Fusion Embedding Transformer (STFEformer) is proposed for traffic flow prediction. Specifically, we propose a fusion embedding layer to capture and fuse both native information and spatial–temporal features, aiming to achieve a comprehensive representation of traffic data characteristics. Then, we introduce a spatial self-attention module designed to enhance detection of dynamic and long-range spatial correlations by focusing on interactions between similar nodes. Extensive experiments conducted on three real-world datasets demonstrate that STFEformer significantly outperforms various baseline models, notably achieving up to a 5.6% reduction in Mean Absolute Error (MAE) on the PeMS08 dataset compared to the next-best model. Furthermore, the results of ablation experiments and visualizations are employed to clarify and highlight our model’s performance. STFEformer represents a meaningful advancement in traffic flow prediction, potentially influencing future research and applications in ITSs by providing a more robust framework for managing and analyzing traffic data.

Predicting and compensating for small-sample thermal information data in precision machine tools: A spatial-temporal interactive integration network and digital twin system approach

The thermal errors in ball screws present complex spatial-temporal characteristics and exhibit persistent long-term memory, significantly affecting the machining accuracy of the whole machine tool in applications. Deep learning has emerged as a promising approach to predict these errors using thermal data. However, the scarcity of extensive thermal datasets, typically due to the prohibitive costs and time required for experimental acquisition, hampers the comprehensive exploitation of spatial-temporal and long-term memory attributes. Moreover, the current thermal error compensation system does not support the deployment of deep learning-based thermal error models due to its weak real-time performance. These limitations lead to suboptimal prediction accuracy and inadequate error compensation. Addressing this challenge, a novel spatial-temporal interactive integration network is proposed, and this is a sophisticated model that synergistically blends a time memory gate and a spatial-temporal fusion gate within an advanced attention-based spatial-temporal graph convolutional framework. Our network ingeniously leverages spatial data to anchor long-term memory while employing temporal data for selective memory feature extraction, facilitating a refined integration of spatial-temporal features. Concurrently, the integration of gated recurrent units and a time attention layer meticulously extracts and enhances temporal features, bolstered by our innovative time memory gate, which is adept at handling small-sample scenarios and refining thermal error predictions. Critically, the digital twin system for thermal error compensation is constructed to improve the system’s real-time performance. The integration of this network into the digital twin system marks a pivotal advancement in thermal error compensation. Our empirical results underscore the system's remarkable robustness and superior predictive accuracy, demonstrating a significant reduction in positioning and machining errors (over 90% and 80% respectively) even with constrained thermal data inputs. These advancements not only delineate a substantial leap in predictive accuracy but also underscore our contribution to reducing operational costs and enhancing the efficacy of machine tools.

Power load forecasting based on spatial–temporal fusion graph convolution network

The cyclical behavior of electricity users facilitates the prediction of future electricity loads through the analysis of historical data, in addition to considering the spatial correlation among various electricity units. Building on these observations, this study transforms the power loads generated by regional-level user units into graph signals for a more effective visualization of spatial–temporal electricity consumption patterns. Specifically, the graph-based model harnesses its unique spatial–temporal dependencies to deduce the dynamics of electricity consumption patterns and enhance the precision of electricity generation predictions. Introducing a novel deep learning model, the Spatial-Temporal Fusion Graph Convolutional Network (FSTGRNN), which integrates temporal and spatial correlations for the prediction of electricity load values at future time steps. FSTGRNN is an end-to-end model that, firstly, delineates the spatial and temporal patterns of electric loads by constructing spatial relationship graphs using the FC-LSTM and GC-GRU models designed within the spatial–temporal block (ST-Block). Secondly, the spatial–temporal fusion block (STF-Block) in FSTGRNN, equipped with a gating mechanism, encodes the graph signals to capture both long-term dependencies and short-term correlations. The resulting integrated representation is utilized for the forecasting process. Experimental validation is conducted on two electricity load datasets, and the forecasting results clearly demonstrate the superiority and efficiency of the proposed deep learning system.

Spatial–Temporal Fusion Gated Transformer Network (STFGTN) for Traffic Flow Prediction

Traffic flow prediction is essential for smart city management and planning, aiding in optimizing traffic scheduling and improving overall traffic conditions. However, due to the correlation and heterogeneity of traffic data, effectively integrating the captured temporal and spatial features remains a significant challenge. This paper proposes a model spatial–temporal fusion gated transformer network (STFGTN), which is based on an attention mechanism that integrates temporal and spatial features. This paper proposes an attention mechanism-based model to address these issues and model complex spatial–temporal dependencies in road networks. The self-attention mechanism enables the model to achieve long-term dependency modeling and global representation of time series data. Regarding temporal features, we incorporate a time embedding layer and a time transformer to learn temporal dependencies. This capability contributes to a more comprehensive and accurate understanding of spatial–temporal dynamic patterns throughout the entire time series. As for spatial features, we utilize DGCN and spatial transformers to capture both global and local spatial dependencies, respectively. Additionally, we propose two fusion gate mechanisms to effectively accommodate to the complex correlation and heterogeneity of spatial–temporal information, resulting in a more accurate reflection of the actual traffic flow. Our experiments on three real-world datasets illustrate the superior performance of our approach.

Filling gaps in cloudy Landsat LST product by spatial-temporal fusion of multi-scale data

Land surface temperature (LST) is an important factor in studies of surface energy fluxes between the Earth's surface and atmosphere. The Landsat LST product has been applied widely due to its fine spatial resolution and high data quality. Frequent cloud coverage, however, results in different degrees of gaps in the Landsat LST images, restricting greatly their application in practical cases requiring continuous LST time-series. Gap filling is an important technique used to reconstruct the missing data in the Landsat LST images. Using only temporally neighboring data from the same sensor, however, current gap filling methods struggle to deal with large spatial gaps and large temporal changes in LST. To solve these challenging issues, in this paper, multi-scale data (i.e., the 2 km, all-sky, hourly GOES LST product) were proposed for gap filling of Landsat LST images, and a spatial-temporal random forest (STRF) method was developed. The GOES LST product can provide effective information on LST in the gaps of the target Landsat LST data, with great potential to decreasing the uncertainty in gap filling when large LST change exists or the gap size is large. Through experiments in six regions of the USA, the proposed STRF method was shown to be more accurate than three typical gap filling and spatio-temporal fusion methods, and particularly advantageous for cases involving large temporal changes or large spatial gaps in LST. Specifically, STRF produces an average root mean square error (RMSE) of 1.6444 K and average correlation coefficient (CC) of 0.8653 for the six regions, which are 0.1133 K smaller and 0.0176 larger than the most competitive benchmark method (i.e., the spatial random forest (SRF) method). Conceptually, by spatial-temporal fusion of multi-scale data in gap filling, STRF provides a new paradigm for both spatial-temporal fusion and gap filling. Furthermore, the proposed STRF has the potential to be applied to various gap filling tasks in other situations.

THE FEASIBILITY OF FUSING SATELLITE IMAGERIES FOR HIGH-FREQUENCY SEA LEVEL MONITORING

Enhancing the capacity to monitor swift environmental shifts at finer scales requires satellite image that offers high spatial and temporal resolution. However, no individual satellite can offer images meeting both criteria simultaneously. To tackle this challenge, spatial temporal fusion algorithms have been developed to derive fine-scale and time-series images. Conversely, effective monitoring of water levels is crucial for preventing natural disaster, such as flood and tsunami mitigation. Yet, monitoring these natural changes regularly poses challenges for remote sensing satellites, given their limitations in either spatial or temporal resolution. For instance, the spatial resolution of 30 meters of Landsat 8 provides imagery with a but lacks the temporal resolution needed to capture dynamic events. On the contrary, the Himawari 8 has the capability to monitor the entire hemisphere every 10 minutes. However, its inadequate resolution affects the precision of sea water change mapping. This research seeks to utilize Landsat OLI and Himawari-8 images jointly for tracking sea level variation patterns. Our approach involves calculating a water index from both Landsat and Himawari images, then using an image fusion algorithm to merge these indices. Next, we identify water coverage by applying a specific threshold on the water index. The comparison of water percentages with reference water height observations has delivered encouraging outcomes.

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

As an emerging neuromorphic camera with an asynchronous working mechanism, spike camera shows good potential for high-speed vision tasks. Each pixel in spike camera accumulates photons persistently and fires a spike whenever the accumulation exceeds a threshold. Such high-frequency fine-granularity photon recording facilitates the analysis and recovery of dynamic scenes with high-speed motion. This paper considers the optical flow estimation problem for spike cameras. Due to the Poisson nature of incoming photons, the occurrence of spikes is random and fluctuating, making conventional image matching inefficient. We propose a Hierarchical Spatial-Temporal (HiST) fusion module for spike representation to pursue reliable feature matching and develop a robust optical flow network, dubbed as HiST-SFlow. The HiST extracts features at multiple moments and hierarchically fuses the spatial-temporal information. We also propose an intra-moment filtering module to further extract the feature and suppress the influence of randomness in spikes. A scene loss is proposed to ensure that this hierarchical representation recovers the essential visual information in the scene. Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared with the existing methods. The source codes are available at https://github.com/ruizhao26/HiST-SFlow.

Spatiotemporal feature learning for no-reference gaming content video quality assessment

Recently, over-the-top live gaming content video (GCV) services have significantly contributed to the overall internet traffic. Consequently, there is a growing demand of GCV quality assessment (GCVQA) to maintain service quality. Although recent literature has proposed a few GCVQA methods, these mainly focus on extracting spatial features and temporal fusion separately, limiting their performance due to the neglect of spatiotemporal feature learning, which is crucial for GCV as it typically shares spatial and temporal features across frames. To address this, we propose a novel GCVQA model, focusing on GCV spatiotemporal feature learning. First, we employ a multi-task self-supervised learning spatiotemporal pyramid convolutional neural network (STP-CNN) model to extract short-term spatiotemporal quality feature representations (STQFR) of GCVs. Our STP-CNN model specifically extracts multiscale spatiotemporal features from various temporal scales of multi-frames in pyramid mode, enabling dynamic learning of diverse spatiotemporal cues. Subsequently, we propose the differential Transformer model to process all short-term STQFR within a GCV, extracting global spatiotemporal features of GCV to assess the overall quality of GCV. To evaluate the effectiveness of our proposed method, we conducted experiments using four GCVQA datasets. The results demonstrate that our method outperforms existing approaches in predicting the perceived quality of GCV.

Underwater acoustic signal classification based on a spatial–temporal fusion neural network

Underwater acoustic signal classification based on a spatial–temporal fusion neural network