Spatio-temporal Contextual Information Research Articles

Precipitation forecasting with radar echo maps based on interactive spatiotemporal context with self-attention and the MIM model

ABSTRACT A sequence of radar echo maps can visually show the motion and variation trends of the echo area, making it a common tool for precipitation forecasting. The spatiotemporal context reveals the correlations of variation trends among different parts within the echo area. This paper proposes a novel precipitation forecasting model, ISTC-SA-MIM (Interactive Spatiotemporal Context Learning with Self-Attention and Memory in Memory), based on the MIM. Leveraging the spatiotemporal interactions and self-attention mechanism of the ISTC-SA structure, the proposed model effectively captures both long-term and short-term spatiotemporal contexts. By memorizing the spatiotemporal context and non-stationary information, ISTC-SA-MIM can accurately predict the motion and variation trends of the echo area. Radar echo data from the Qingdao station are collected as the dataset to evaluate the commonly used spatiotemporal models and ISTC-SA-MIM. The experiments demonstrate that ISTC-SA-MIM can predict the variation trends of the echo area more accurately by learning the spatiotemporal context.

Sequential Fusion Based Multi-Granularity Consistency for Space-Time Transformer Tracking

Regarded as a template-matching task for a long time, visual object tracking has witnessed significant progress in space-wise exploration. However, since tracking is performed on videos with substantial time-wise information, it is important to simultaneously mine the temporal contexts which have not yet been deeply explored. Previous supervised works mostly consider template reform as the breakthrough point, but they are often limited by additional computational burdens or the quality of chosen templates. To address this issue, we propose a Space-Time Consistent Transformer Tracker (STCFormer), which uses a sequential fusion framework with multi-granularity consistency constraints to learn spatiotemporal context information. We design a sequential fusion framework that recombines template and search images based on tracking results from chronological frames, fusing updated tracking states in training. To further overcome the over-reliance on the fixed template without increasing computational complexity, we design three space-time consistent constraints: Label Consistency Loss (LCL) for label-level consistency, Attention Consistency Loss (ACL) for patch-level ROI consistency, and Semantic Consistency Loss (SCL) for feature-level semantic consistency. Specifically, in ACL and SCL, the label information is used to constrain the attention and feature consistency of the target and the background, respectively, to avoid mutual interference. Extensive experiments have shown that our STCFormer outperforms many of the best-performing trackers on several popular benchmarks.

Estimating daily semantic segmentation maps of classified ocean eddies using sea level anomaly data from along-track altimetry.

Mesoscale eddies, which are fast-moving rotating water bodies in the ocean with horizontal scales ranging from 10 km to 100 km and above, are considered to be the weather of the oceans. They are of interest to marine biologists, oceanographers, and geodesists for their impact on water mass, heat, and nutrient transport. Typically, gridded sea level anomaly maps processed from multiple radar altimetry missions are used to detect eddies. However, multi-mission sea level anomaly maps obtained by the operational processors have a lower effective spatiotemporal resolution than their grid spacing and temporal resolution, leading to inaccurate eddy detection. In this study, we investigate the use of higher-resolution along-track sea level anomaly data to infer daily two-dimensional segmentation maps of cyclonic, anticyclonic, or non-eddy areas with greater accuracy than using processed sea level anomaly grid map products. To tackle this challenge, we propose a deep neural network that uses spatiotemporal contextual information within the modality of along-track data. This network is capable of producing a two-dimensional segmentation map from data with varying sparsity. We have developed an architecture called Teddy, which uses a Transformer module to encode and process spatiotemporal information, and a sparsity invariant CNN to infer a two-dimensional segmentation map of classified eddies from the ground tracks of varying sparsity on the considered region. Our results show that Teddy creates two-dimensional maps of classified eddies from along-track data with higher accuracy and timeliness when compared to commonly used methods that work with less accurate preprocessed sea level anomaly grid maps. We train and test our method with a carefully curated and independent dataset, which can be made available upon request.

Automatic detection of myocardial ischaemia using generalisable spatio-temporal hierarchical Bayesian modelling of DCE-MRI

Dynamic Contrast Enhanced Magnetic Resonance Imaging (DCE-MRI) can be used as a non-invasive method for the assessment of myocardial perfusion. The acquired images can be utilised to analyse the spatial extent and severity of myocardial ischaemia (regions with impaired microvascular blood flow). In the present paper, we propose a novel generalisable spatio-temporal hierarchical Bayesian model (GST-HBM) to automate the detection of ischaemic lesions and improve the in silico prediction accuracy by systematically integrating spatio-temporal context information. We present a computational inference procedure with an adequate trade-off between accuracy and computational efficiency, whereby model parameters are sampled from the posterior distribution with Gibbs sampling, while lower-level hyperparameters are selected using model selection strategies based on the Watanabe Akaike information criterion (WAIC). We have assessed our method on both synthetic (in silico) data with known gold-standard and 12 sets of clinical first-pass myocardial perfusion DCE-MRI datasets. We have also carried out a comparative performance evaluation with four established alternative methods: Gaussian mixture model (GMM), opening and closing operations based on Gaussian mixture model (GMMC&Omax), Markov random field constrained Gaussian mixture model (GMM-MRF) and model-based hierarchical Bayesian model (M-HBM). Our results show that the proposed GST-HBM method achieves much higher in silico prediction accuracy than the established alternative methods. Furthermore, this method appears to provide a more robust delineation of ischaemic lesions in datasets affected by spatially variant noise.

Self-Supervised Spatiotemporal Masking Strategy-Based Models for Traffic Flow Forecasting

Traffic flow forecasting is an important function of intelligent transportation systems. With the rise of deep learning, building traffic flow prediction models based on deep neural networks has become a current research hotspot. Most of the current traffic flow prediction methods are designed from the perspective of model architectures, using only the traffic features of future moments as supervision signals to guide the models to learn the spatiotemporal dependence in traffic flow. However, traffic flow data themselves contain rich spatiotemporal features, and it is feasible to obtain additional self-supervised signals from the data to assist the model to further explore the underlying spatiotemporal dependence. Therefore, we propose a self-supervised traffic flow prediction method based on a spatiotemporal masking strategy. A framework consisting of symmetric backbone models with asymmetric task heads were applied to learn both prediction and spatiotemporal context features. Specifically, a spatiotemporal context mask reconstruction task was designed to force the model to reconstruct the masked features via spatiotemporal context information, so as to assist the model to better understand the spatiotemporal contextual associations in the data. In order to avoid the model simply making inferences based on the local smoothness in the data without truly learning the spatiotemporal dependence, we performed a temporal shift operation on the features to be reconstructed. The experimental results showed that the model based on the spatiotemporal context masking strategy achieved an average prediction performance improvement of 1.56% and a maximum of 7.72% for longer prediction horizons of more than 30 min compared with the backbone models.

Infrared Small Target Detection Based on a Temporally-Aware Fully Convolutional Neural Network

In the field of computer vision, the detection of infrared small targets (IRSTD) is a crucial research area that plays an important role in space exploration, infrared warning systems, and other applications. However, the existing IRSTD methods are prone to generating a higher number of false alarms and an inability to accurately locate the target, especially in scenarios with low signal-to-noise ratio or high noise interference. To address this issue, we proposes a fully convolutional-based small target detection algorithm (FCST). The algorithm builds on the anchor-free detection method FCOS and adds a focus structure and a single aggregation approach to design a lightweight feature extraction network that efficiently extracts features for small targets. Furthermore, we propose a feature refinement mechanism to emphasize the target and suppress conflicting information at multiple scales, enhancing the detection of infrared small targets. Experimental results demonstrate that the proposed algorithm achieves a detection rate of 95% and a false alarm rate of 2.32% for IRSTD tasks. To tackle even more complex scenarios, we propose a temporally-aware fully convolutional infrared small target detection (TFCST) algorithm that leverages both spatial and temporal information from sequence images. Building on a single-frame detection network, the algorithm incorporates ConvLSTM units to extract spatiotemporal contextual information from the sequence images, boosting the detection of infrared small targets. The proposed algorithm shows fast detection speed and achieves a 2.73% improvement in detection rate and an 8.13% reduction in false alarm rate relative to the baseline single-frame detection networks.

MS-LSTM: Exploring spatiotemporal multiscale representations in video prediction domain

The drastic variation of motion in spatial and temporal dimensions makes the video prediction task extremely challenging. Existing RNN models obtain higher performance by deepening or widening the model. They obtain the multi-scale features of the video only by stacking layers, which is inefficient and brings unbearable training costs (such as memory, FLOPs, and training time). Different from them, this paper proposes a spatiotemporal multi-scale model called MS-LSTM wholly from a multi-scale perspective. On the basis of stacked layers, MS-LSTM incorporates two additional efficient multi-scale designs to fully capture spatiotemporal context information. Concretely, we employ LSTMs with mirrored pyramid structures to construct spatial multi-scale representations and LSTMs with different convolution kernels to construct temporal multi-scale representations. We theoretically analyze the training cost and performance of MS-LSTM and its components. Detailed comparison experiments with twelve baseline models on four video datasets show that MS-LSTM has better performance but lower training costs.

STAB-GCN: A Spatio-Temporal Attention-Based Graph Convolutional Network for Group Activity Recognition

Group activity recognition is a central theme in many domains, such as sports video analysis, CCTV surveillance, sports tactics, and social scenario understanding. However, there are still challenges in embedding actors’ relations in a multi-person scenario due to occlusion, movement, and light. Current studies mainly focus on collective and individual local features from the spatial and temporal perspectives, which results in inefficiency, low robustness, and low portability. To this end, a Spatio-Temporal Attention-Based Graph Convolution Network (STAB-GCN) model is proposed to effectively embed deep complex relations between actors. Specifically, we leverage the attention mechanism to attentively explore spatio-temporal latent relations between actors. This approach captures spatio-temporal contextual information and improves individual and group embedding. Then, we feed actor relation graphs built from group activity videos into our proposed STAB-GCN for further inference, which selectively attends to the relevant features while ignoring those irrelevant to the relation extraction task. We perform experiments on three available group activity datasets, acquiring better performance than state-of-the-art methods. The results verify the validity of our proposed model and highlight the obstructive impacts of spatio-temporal attention-based graph embedding on group activity recognition.

Modeling Long- and Short-Term User Preferences via Self-Supervised Learning for Next POI Recommendation

With the accumulation of check-in data from location-based services, next Point-of-Interest (POI) recommendations are gaining increasing attention. It is well known that the spatio-temporal contextual information of user check-in behavior plays a crucial role in handling vital and inherent challenges in next POI recommendation, including capture of user dynamic preferences and the sparsity problem of check-in data. However, many studies either ignore or simply stack the context features with the embedding of POIs while relying only on POI recommendation loss to optimize the entire model, therefore failing to take full advantage of the potential information in contexts. Additionally, users’ interests are usually unstable and evolve over time, and accordingly recent studies have proposed various approaches to predict users’ next POIs by incorporating contextual information and modeling both their long- and short-term preferences, respectively. Yet many studies overemphasize the final POI recommendation performance, and the association between POI sequences and contextual information is not well embodied in data representations. In this article, we focus on the preceding problems and propose a unified attention framework for next POI recommendation by modeling users’ Long- and Short-term Preferences via Self-supervised Learning (LSPSL). Specifically, based on the self-attention network and two self-supervised optimization objectives, LSPSL first deeply exploits the intrinsic correlations between POI sequences and contextual information through pre-training, which strengthens data representations. Then, supported by pre-trained contextualized embeddings, LSPSL models and fuses users’ complex long- and short-term preferences in a unified way. Extensive experiments on real-world datasets demonstrate the superiority of our model compared with other state-of-the-art approaches.

Spatial Context-Aware Time-Series Forecasting for QoS Prediction

With the explosive growth of web services, the increasing sparse and dynamic QoS (quality of service) data pose a great challenge to QoS prediction in service recommendation. How to fully utilize the contextual information of service invocation and their potential relationships becomes the key to improving the accuracy of QoS prediction. In this paper, a spatial context-aware time series forecasting (SCATSF) framework is proposed for QoS prediction by considering both the temporal and the spatial context of users and services. SCATSF has three main parts: time-aware neighbors selection, spatial context-aware interaction learning and time series forecasting for QoS prediction. First, a novel time series similarity (TSS) is proposed to measure the similarity of users or services based on time-varying QoS fluctuation, in order to select appropriate neighbors for the target user or service. Furthermore, the contextual information of neighbors, as well as the contextual information of users and services, are both integrated to enrich the spatial context of service invocation, and a pairwise multi-layer deep network (PMLDN) is developed to consolidate multiple features and to learn feature interactions in pairs. With the help of spatial contextual information and time-aware QoS information, a spatial context-aware GRU model (SCA-GRU) is finally present to complete the time series forecasting for QoS prediction. Experimental results on the prediction of response time in a real-world dataset demonstrate that our SCATSF approach can effectively utilize spatio-temporal contextual information, so that it can achieve higher accuracy than many existing methods.

Real-time and robust visual tracking with scene-perceptual memory

Unmanned aerial vehicle (UAV) based aerial visual tracking is one of the research hotspots in computer vision. However, the mainstream trackers for UAV still have two shortcomings: (1) the accuracy of correlation filter tracker is greatly improved with more complex model, it impedes accuracy-speed trade-off. (2) object occlusion and camera motion in the aerial tracking scene also seriously restrict the application of aerial tracking. To address these problems, and inspired by AutoTrack tracker, we propose an aerial correlation filtering tracker based on scene-perceptual memory, Fast-AutoTrack. Firstly, to perceive and judge tracking anomalies, such as object occlusion and camera motion, inspired by the peak sidelobe ratio and AutoTrack, a confidence score is designed by perceiving and remembering the changing trend of the confidence and the local historical confidence. Secondly, after tracking anomaly occurring, several search regions are predicted based on the local object motion trend and the Spatio-temporal context information for object re-detection. Finally, to accelerate the model updating, the perceptual hashing algorithm (PHA) is used to obtain the similarity of the search regions between two adjacent frames. On typical aerial tracking datasets UAVDT, UAV123@10fps, and DTB70, Fast-AutoTrack run 71.4% faster than AutoTrack with almost equal accuracy and show favorable accuracy-speed trade-off.

A transformer model for learning spatiotemporal contextual representation in fMRI data.

Representation learning is a core component in data-driven modeling of various complex phenomena. Learning a contextually informative representation can especially benefit the analysis of fMRI data because of the complexities and dynamic dependencies present in such datasets. In this work, we propose a framework based on transformer models to learn an embedding of the fMRI data by taking the spatiotemporal contextual information in the data into account. This approach takes the multivariate BOLD time series of the regions of the brain as well as their functional connectivity network simultaneously as the input to create a set of meaningful features that can in turn be used in various downstream tasks such as classification, feature extraction, and statistical analysis. The proposed spatiotemporal framework uses the attention mechanism as well as the graph convolution neural network to jointly inject the contextual information regarding the dynamics in time series data and their connectivity into the representation. We demonstrate the benefits of this framework by applying it to two resting-state fMRI datasets, and provide further discussion on various aspects and advantages of it over a number of other commonly adopted architectures.

TCIANet: Transformer-Based Context Information Aggregation Network for Remote Sensing Image Change Detection

Change detection based on remote sensing data is an important method to detect the earth surface changes. With the development of deep learning, convolutional neural networks have excelled in the field of change detection. However, the existing neural network models are susceptible to external factors in the change detection process, leading to pseudo change and missed detection in the detection results. In order to better achieve the change detection effect and improve the ability to discriminate pseudo-change, this paper proposes a new method, namely, transformer-based context information aggregation network (TCIANet) for remote sensing image change detection. First, we use a filter-based visual tokenizer (FVT) to segment each temporal feature map into multiple visual semantic tokens. Second, the addition of the progressive sampling vision transformer (PS-ViT) not only effectively excludes the interference of irrelevant changes, but also uses the transformer encoder to obtain compact spatio-temporal context information in the token set. Then, the tokens containing rich semantic information are fed into the pixel space and the transformer decoder is used to acquire pixel-level features. In addition, we use the feature fusion module (FFM) to fuse low-level semantic feature information to complete the extraction of coarse contour information of the changed region. Then the semantic relationships between object regions and contours are captured by contour-graph reasoning module (CGRM) to obtain feature maps with complete edge information. Finally, the prediction model is used to discriminate the change of feature information and generate the final change map. Numerous experimental results show that our method has more obvious advantages in visual effect and quantitative evaluation than other methods.

Spatio-temporal fire detection based on brightness temperature change in Himawari-8 images

ABSTRACT The geostationary satellite Himawari-8 has become the primary source for fire remote-sensing detection because of its advantages of high frequency, large width and easy access. To solve the problem of difficulty in small fire detection and with a high omission on Himawari-8 image, a spatio-temporal fire detection (STFD) method based on Himawari-8 image brightness temperature change is proposed. Calculate the mean value difference of brightness temperature between multi-time series images before the target time, and use the least square model to fit the ideal mean value difference of brightness temperature between the target time and the image at the previous time. Then, the difference value of the actual brightness temperature between the target time and the last image time is calculated. The initial brightness temperature change pixels are obtained by the comparative analysis method. Based on the spatial statistical characteristics, the potential fire point is determined by the brightness temperature value ratio of the image, temperature mean and standard temperature deviation. Then, combined with the spatio-temporal context information, the persistent fire points are detected according to the threshold conditions to supplement undetected persistent fires. The results of experiments on the Himawari-8 images in Guizhou, China and southwestern Australia indicate that STFD exhibits superior performance on small fire detection points, and the accuracy is better than the several traditional methods.

SC-LPR: Spatiotemporal context based LiDAR place recognition

• A spatiotemporal and semantic information fusion based LiDAR place recognition method improves recognition accuracy. • A novel feature encoding module is proposed to effectively aggregate spatiotemporal features. • Spatiotemporal information is fused without geometrical transformations between LiDAR scans. • A superior feature similarity evaluation network is proposed to predict similarity score. 3D LiDAR-based place recognition remains challenging due to ambiguities and dynamics of scenes. Place representation can be learned effectively by deep learning, but perceptual confusion is inevitable based on single-scan. To alleviate the problem, we propose a new place recognition method named SC-LPR. In this method, spatiotemporal contextual information from LiDAR scans is used to increase the capacity of features’ representation. Firstly, a semantic graph is constructed to represent the topological geometric map of each LiDAR scan. Then, an end-to-end network is designed to predict similarity, in which GRU-EdgeConv++ is proposed to learn discriminative spatiotemporal feature representation and a novel C&TNT to predict a score. We evaluate our approach on the KITTI odometry benchmark. The experimental results show that our method can effectively fuse spatiotemporal information for place recognition. Compared with the state-of-the-art methods, our approach is superior on the KITTI dataset and can achieve competitive performance. To benefit the community by serving as a benchmark for place recognition, the code of our method will be made open-source on Github 1 1 https://github.com/Daideyun/SC-LPR .

User Model-Based Personalized Recommendation Algorithm for News Media Education Resources

Traditional recommendations for news and media education resources usually ignore the importance of sequential patterns in user check-in behavior and fail to effectively capture the complex and dynamically changing interests of users. As a result, this study provides a recommendation model for news and media education materials based on a user model. To capture changes in users’ interests, the model can represent and fuse short-term and long-term preferences separately. For short-term preferences, a long- and short-term memory network incorporating spatiotemporal contextual information is proposed to learn complex sequential transfer patterns in users’ check-in behaviors and further extract short-term preferences accurately through a goal-based attention mechanism. A user attention-based approach is utilized to capture fine-grained links between users and interest points for long-term preferences. Finally, experimental simulations are conducted on two datasets, Foursquare and Gowalla. The results show that the proposed user model-based recommendation model for news media education resources has better performance compared with the mainstream recommendation methods on different evaluation criteria, which validates the effectiveness of the proposed model.

An Attention-Based Spatiotemporal GGNN for Next POI Recommendation

The task of Point-of-Interest (POI) recommendation is to recommend the next interest locations for users. Gated Graph Neural Network (GGNN) has been proved to be effective on POI recommendation tasks. However, existing GGNN solutions rarely consider the spatiotemporal information between nodes in the sequence graph, which is essential for modeling user check-in behaviors in next POI recommendation. In this paper, we propose an attention-based spatiotemporal gated graph neural network model (ATST-GGNN) for next POI recommendation. Firstly, the user’s check-in sequence is represented as a graph structure. Secondly, we use spatiotemporal context information to dynamically update nodes in the sequence graph, and obtain the complex transfer relationships between the check-ins. Thirdly, each session is then represented as the composition of the long and short preference using an attention network. However, current short preference fails to model union-level sequential patterns, we improve the local embedding representation of graph nodes by window pooling method, as well as the global embedding representation of graph nodes by integrating it into attention mechanism. Finally, the objective function is constructed by cross entropy and the model parameters are learned. The experimental results show that the precision rate and mean reciprocal ranking of ATST-GGNN method are greatly improved compared with the state-of-art methods. It has good application prospect.

Fusing User Preferences and Spatiotemporal Information for Sequential Recommendation

At present, the research on sequence recommendation mainly focuses on using the historical interaction data between users and items to mine their relationship, so as to predict the next interaction between users and items, then generate the personalized recommendation. Spatiotemporal information is very important to further improve the accuracy and quality of recommendation, but the existing sequence recommendation models are mainly based on recurrent neural network (RNN), and pay less attention to spatiotemporal information. Most of the recommendation models are still in the early stage of merging spatiotemporal context information, and the processing effect of long sequence data is not ideal. a sequential recommendation model integrating user preferences and spatiotemporal information is proposed. The model captures user item long-term preferences through spatiotemporal GRU algorithm and user item short-term preferences through attention mechanism. Finally, the learned long-term and short-term preference features and user portrait features are combined to predict the next recommendation location. The experimental results on two real data sets Foursquare and Brightkite show that the proposed model performs better than state-of-the-arts in three evaluation indicators HR@K, NDCG@K and MAP@K.

An Attention-Based Spatiotemporal LSTM Network for Next POI Recommendation

Next point-of-interest (POI) recommendation, also known as a natural extension of general POI recommendation, is recently proposed to predict user's next destination and has attracted considerable research interest. It focuses on learning users’ sequential patterns of check-in behavior and on training personalized recommendation models using different types of contextual information. Unfortunately, most of the previous studies failed to incorporate the spatiotemporal contextual information, which plays a critical role in analyzing user check-in behavior, into recommending the next POI. In recent years, embedding learning and recurrent neural network (RNN) based approaches show promising performance for modeling sequential patterns of check-in behavior in next POI recommendation. However, not all of the historical check-in records contribute equally to the next-step check-in behavior. To provide better next POI recommendation performance, we first proposed a spatiotemporal long and short-term memory (ST-LSTM) network. By feeding the spatiotemporal contextual information into the LSTM network in each step, ST-LSTM can model the spatial and temporal information better. Also, we developed an attention-based spatiotemporal LSTM (ATST-LSTM) network for next POI recommendation. By using the attention mechanism, ATST-LSTM can focus on the relevant historical check-in records in a check-in sequence selectively using the spatiotemporal contextual information. Besides, we conducted a comprehensive performance evaluation using large-scale real-world datasets collected from two popular location-based social networks, namely Gowalla and Brightkite. Experimental results indicated that the proposed ATST-LSTM network outperformed two state-of-the-art next POI recommendation approaches regarding three commonly-used evaluation metrics.

Target Adaptive Tracking Based on GOTURN Algorithm with Convolutional Neural Network and Data Fusion.

With the advent of the artificial intelligence era, target adaptive tracking technology has been rapidly developed in the fields of human-computer interaction, intelligent monitoring, and autonomous driving. Aiming at the problem of low tracking accuracy and poor robustness of the current Generic Object Tracking Using Regression Network (GOTURN) tracking algorithm, this paper takes the most popular convolutional neural network in the current target-tracking field as the basic network structure and proposes an improved GOTURN target-tracking algorithm based on residual attention mechanism and fusion of spatiotemporal context information for data fusion. The algorithm transmits the target template, prediction area, and search area to the network at the same time to extract the general feature map and predicts the location of the tracking target in the current frame through the fully connected layer. At the same time, the residual attention mechanism network is added to the target template network structure to enhance the feature expression ability of the network and improve the overall performance of the algorithm. A large number of experiments conducted on the current mainstream target-tracking test data set show that the tracking algorithm we proposed has significantly improved the overall performance of the original tracking algorithm.