Spatial-temporal Research Articles

Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection.

Besides combining appearance and motion information, another crucial factor for video salient object detection (VSOD) is to mine spatial-temporal (ST) knowledge, including complementary long-short temporal cues and global-local spatial context from neighboring frames. However, the existing methods only explored part of them and ignored their complementarity. In this article, we propose a novel complementary ST transformer (CoSTFormer) for VSOD, which has a short-global branch and a long-local branch to aggregate complementary ST contexts. The former integrates the global context from the neighboring two frames using dense pairwise attention, while the latter is designed to fuse long-term temporal information from more consecutive frames with local attention windows. In this way, we decompose the ST context into a short-global part and a long-local part and leverage the powerful transformer to model the context relationship and learn their complementarity. To solve the contradiction between local window attention and object motion, we propose a novel flow-guided window attention (FGWA) mechanism to align the attention windows with object and camera movements. Furthermore, we deploy CoSTFormer on fused appearance and motion features, thus enabling the effective combination of all three VSOD factors. Besides, we present a pseudo video generation method to synthesize sufficient video clips from static images for training ST saliency models. Extensive experiments have verified the effectiveness of our method and illustrated that we achieve new state-of-the-art results on several benchmark datasets.

Handling protracted crises: Communication challenges, constraints and opportunities

AbstractRecent research on crisis communication has focused on the challenges of complex and protracted crises. In this essay, it is argued that such crises must be handled communicatively by putting the communication challenges, constraints and opportunities at the centre. To this end, the combination of the notions of the rhetorical situation and kairos is suggested as a keystone that provides a bottom‐up, context‐sensitive approach needed to cope with the fluid and changing situations of such crisis. Importantly, the life cycle of a protracted crisis can not only be understood in terms of physical factors (e.g., infection rates in a pandemic) but is also formed through the public's perceptions and the communicative choices of crisis communicators calling for action to mitigate or end a crisis. The rhetorical situation and kairos highlight the importance of spatial temporality, the uniqueness of the situations and the constitutive dimension of communication in a crisis. This implies the identification of the opportune moments to present a response and recognising the important ‘spaces’ between typical crisis situations.

Assessment of land use and land cover change detection and prediction using deep learning techniques for the southwestern coastal region, Goa, India.

Understanding the connections between human activities and the natural environment depends heavily on information about land use and land cover (LULC) in the form of accurate LULC maps. Environmental monitoring using deep learning (DL) is rapidly growing to preserve a sustainable environment in the long term. For establishing effective policies, regulations, and implementation, DL can be a valuable tool for assessing environmental conditions and natural resources that will positively impact the ecosystem. This paper presents the assessment of land use and land cover change detection (LULCCD) and prediction using DL techniques for the southwestern coastal region, Goa, also known as the tourist destination of India. It consists of three components: (i) change detection (CD), (ii) quantification of LULC changes, and (iii) prediction. A new CD assessment framework, Spatio-Temporal Encoder-Decoder Self Attention Network (STEDSAN), is proposed for the LULCCD process. A dual branch encoder-decoder network is constructed using strided convolution with downsampling for the encoder and transpose convolution with upsampling for the decoder to assess the bitemporal images spatially. The self-attention (SA) mechanism captures the complex global spatial-temporal (ST) interactions between individual pixels over space-time to produce more distinct features. Each branch accepts the LULC map of 2 years as one of its inputs to determine binary and multiclass changes among the bitemporal images. The STEDSAN model determines the patterns, trends, and conversion from one LULC type to another for the assessment period from 2005 to 2018. The binary change maps were also compared with the existing state of the art (SOTA) CD methods, with STEDSAN having an overall accuracy of 94.93%. The prediction was made using an recurrent neural network (RNN) known as long short term memory network (LSTM) for the year 2025. Experiments were conducted to determine area-wise changes in several LULC classes, such as built-up (BU), crops (kharif crop (KC), rabi crop (RC), zaid crop (ZC), double/triple (D/T C)), current fallow (CF), plantation (PL), forests (evergreen forest (EF), deciduous forest (DF), degraded/scurb forest (D/SF) ), littoral swamp (LS), grassland (GL), wasteland (WL), waterbodies max (Wmx), and waterbodies min (Wmn). As per the analysis, over the period of 13 years, there has been a net increase in the amount of BU (1.25%), RC (1.17%), and D/TC( 2.42%) and a net decrease in DF (3.29%) and WL(1.44%) being the most dominant classes being changed. These findings will offer a thorough description of identifying trends in coastal areas that may incorporate methodological hints for future studies. This study will also promote handling the spatial and temporal complexity of remotely sensed data employed in categorizing the coastal LULC of a heterogeneous landscape.

Fully-Connected Spatial-Temporal Graph for Multivariate Time-Series Data

Multivariate Time-Series (MTS) data is crucial in various application fields. With its sequential and multi-source (multiple sensors) properties, MTS data inherently exhibits Spatial-Temporal (ST) dependencies, involving temporal correlations between timestamps and spatial correlations between sensors in each timestamp. To effectively leverage this information, Graph Neural Network-based methods (GNNs) have been widely adopted. However, existing approaches separately capture spatial dependency and temporal dependency and fail to capture the correlations between Different sEnsors at Different Timestamps (DEDT). Overlooking such correlations hinders the comprehensive modelling of ST dependencies within MTS data, thus restricting existing GNNs from learning effective representations. To address this limitation, we propose a novel method called Fully-Connected Spatial-Temporal Graph Neural Network (FC-STGNN), including two key components namely FC graph construction and FC graph convolution. For graph construction, we design a decay graph to connect sensors across all timestamps based on their temporal distances, enabling us to fully model the ST dependencies by considering the correlations between DEDT. Further, we devise FC graph convolution with a moving-pooling GNN layer to effectively capture the ST dependencies for learning effective representations. Extensive experiments show the effectiveness of FC-STGNN on multiple MTS datasets compared to SOTA methods. The code is available at https://github.com/Frank-Wang-oss/FCSTGNN.

Spatial–temporal feature-based End-to-end Fourier network for 3D sign language recognition

Most dynamic sign word misclassifications are caused by redundant spatial–temporal (SPT) feature pruning that lacks language semantic and temporal dependencies. SPT feature recognition is one of the important aspects for the evaluation of the misclassification of dynamic sign words. The redundant pruning of SPT feature space influences the language model of sign confusion, model complexity, and SPT feature similarity. The purpose of this article is to develop a new multi-scale SPT feature-based dynamic sign word recognition approach via a low-cost feature selection method (FS) and End-to-end Fourier convolution neural network (EFCNN). Instead of a sensor fusion technique for obtaining frame position alignment, in the EFCNN, new 3D frame position and coordinates are determined using a pixel weighting and alignment function of the first and succeeding 25 spatial intensities of the 3D video changes across hand motion. The new spatial weight and the original spatial coordinates are fused and truncated in the Fourier domain. We generate the temporal dependence of the fused features. A feature selection known as the FS-EFCNN is introduced to select compact features with a preserved language meaning. Five state-of-the-art feature selection methods, namely Infinite FS (InFS), Relief FS, Fisher, MIM, ILFS, and ensemble FS-EFCNN were deployed to guide and optimize the learning performance of EFCNN. The experimental result analysis highlighted the improved results of the FS-EFCNN method with the best accuracy of 99.86%, 99.89%, and 90.69% on 3D American Sign Language, British Sign Language, and Greek Sign Language data sets, respectively.

Deep Reinforcement Learning for Voltage Control and Renewable Accommodation Using Spatial-Temporal Graph Information

Renewable energy resources (RERs) have been increasingly integrated into distribution networks (DNs) for decarbonization. However, the variable nature of RERs introduces uncertainties to DNs, frequently resulting in voltage fluctuations that threaten system security and hamper the further adoption of RERs. To incentivize more RER penetration, we propose a deep reinforcement learning (DRL)-based strategy to dynamically balance the trade-off between voltage fluctuation control and renewable accommodation. To further extract multi-time-scale spatial-temporal (ST) graphical information of a DN, our strategy draws on a multi-grained attention-based spatial-temporal graph convolution network (MG-ASTGCN), consisting of ST attention mechanism and ST convolution to explore the node correlations in the spatial and temporal views. The continuous decision-making process of balancing such a trade-off can be modeled as a Markov decision process optimized by the deep deterministic policy gradient (DDPG) algorithm with the help of the derived ST information. We validate our strategy on the modified IEEE 33, 69, and 118-bus radial distribution systems, with outcomes significantly outperforming the optimization-based benchmarks. Simulations also reveal that our developed MG-ASTGCN can to a great extent accelerate the convergence speed of DDPG and improve its performance in stabilizing node voltage in an RER-rich DN. Moreover, our method improves the DN's robustness in the presence of generator failures.

MG-ASTN: Multigraph Framework With Attentive Spatial–Temporal Networks for Crowd Mobility Prediction

Predicting urban crowd patterns/flows is a challenging task due to complex spatial-temporal (ST) dependencies. In this paper, we aim to examine and report the capability and effectiveness of the current most widely used Graph Convolutional Networks (GCNs) on mobile data analysis in ST networks for crowd flow forecasting. Specifically, we propose a novel dual-stream framework leveraging Multi-Graph with Attentive Spatial-Temporal Networks (MG-ASTN) to simultaneously predict crowd in-out flow and OD flow based on the trajectory data collected by on-board devices (e.g., GPS). MG-ASTN utilizes Multi-Graph Convolutional Networks encoding non-Euclidean correlations to explore pair-wise relationships among regions. In addition, we further apply a cross-channel attention mechanism with 3D Temporal Convolutional Network to address the heterogeneity of ST features and capture more meaningful data representations for multi-task learning. In the evaluation, we conduct experiments based on 2 real-world datasets and verify most well-known state-of-the-art methods for crowd flow prediction. The results demonstrate that MG-ASTN could outperform other solutions -in-out flow prediction with lowest RMSE and MAE, and OD flow prediction beyond others in most cases, thus it has great potential in modeling the complex correlations among regions in ST networks and enabling accurate prediction in urban computing.

Beyond lawfare: An analysis of law's temporality through Russian‐doll urbanization from Turkey

AbstractThis article opens a discussion about how temporalities in spatial and legal spheres are interlinked and shape both policymaking and governance mechanisms and resistance practices. Taking a case study from Eskişehir, Turkey, the research examines several urban renewal attempts of a municipality on the same urban lands over two decades that used different laws and policy tools in each case while all of which were annulled by court suits. From a perspective of legal anthropology, the analysis shows the limits of lawfare discussions that remain incapable of explaining cases that cannot be categorized with domination and resistance. Instead of focusing on detecting who wins or loses, the article claims that such complex cases could be understood better by scrutinizing temporal dynamics: how do legal, spatial, and social temporalities intertwine and impact policymaking? More specifically, how does the law's temporality (re)shape urbanization? Even further, what political work do temporalities generate? The article offers Russian‐doll urbanization as an analogy and ethnographic metaphor to examine several layers and endlessness of renewal initiatives within the broader process of urbanization. Studying the revelation of each layer unravels entangled temporalities of law and socio‐spatial dynamics and their consequences in policymaking and resistance.

Data Fusion in Infrastructure-Augmented Autonomous Driving System: Why? Where? and How?

This paper is the first to provide a thorough system design overview along with the fusion methods selection criteria of a real-world cooperative autonomous driving system enabled by the Internet of Things (IoT), named Infrastructure-Augmented Autonomous Driving (IAAD). We present an in-depth introduction to the IAAD hardware and software on both road-side and vehicle-side. We extensively characterize the IAAD system and observe that the network condition fluctuation along the road is the main roadblock for cooperative autonomous driving. To address this challenge, we propose new fusion methods, dubbed "inter-frame fusion" and "planning fusion" to complement the state-of-the-art "intra-frame fusion". We demonstrate that each fusion method has its own benefit and constraint. In order to select the best fusion method under varying network conditions, we propose ‘fusion criteria’ to instruct the IAAD system to intelligently make the selection and implement a system framework named adaptive spatial-temporal (S-T) choice to realize the adaptive fusion guided by the ’fusion criteria’. Our real-world field data verifies that S-T choice has significantly improved autonomous driving’s safety and reliability by decreasing the fusion miss ratio from 30% to 7% and remain the planning displacement error within the 1.7m instead of 4m when the network condition exacerbates.

A people-centred framework for exploring water, energy and food security in a small developing island

Small developing islands face a number of environmental and social pressures which impact resource security. This study uses a people-centred framework to investigate social-ecological interactions for water, energy and food security. Ten semi-structured focus group discussions were conducted in Pemba and Unguja islands with village elders and leaders. Results demonstrate that shocks and stresses affecting resource security are attributed to land use and resource competition, deforestation, climate change and insufficient resource infrastructure. The scale and strength of such pressures are heightened in dry seasons and also correspond with spatial characteristics such as remoteness, intensity of land use and amount of natural resource capital. Whilst a number of adaptive responses are identified, these appear to be incremental and do not address the scale of the challenge. Maladaptive responses are also identified; most concerning is the use of poor quality water when piped water was disrupted, reduced nutritional intake during dry season and using unsustainable supplies or methods of obtaining of fuelwood. Findings illustrate the importance of using people-centred approaches for understanding the complexity of social-ecological interactions for resource security. They also demonstrate that interventions for resource management need to consider spatial heterogeneity and temporality in terms of how specific land cover uses connect to differential pressures and adaptation capacity over time.

Enhanced Adjacency Matrix-Based Lightweight Graph Convolution Network for Action Recognition.

Graph convolutional networks (GCNs), which extend convolutional neural networks (CNNs) to non-Euclidean structures, have been utilized to promote skeleton-based human action recognition research and have made substantial progress in doing so. However, there are still some challenges in the construction of recognition models based on GCNs. In this paper, we propose an enhanced adjacency matrix-based graph convolutional network with a combinatorial attention mechanism (CA-EAMGCN) for skeleton-based action recognition. Firstly, an enhanced adjacency matrix is constructed to expand the model's perceptive field of global node features. Secondly, a feature selection fusion module (FSFM) is designed to provide an optimal fusion ratio for multiple input features of the model. Finally, a combinatorial attention mechanism is devised. Specifically, our spatial-temporal (ST) attention module and limb attention module (LAM) are integrated into a multi-input branch and a mainstream network of the proposed model, respectively. Extensive experiments on three large-scale datasets, namely the NTU RGB+D 60, NTU RGB+D 120 and UAV-Human datasets, show that the proposed model takes into account both requirements of light weight and recognition accuracy. This demonstrates the effectiveness of our method.

Easy Begun Is Half Done: Spatial-Temporal Graph Modeling with ST-Curriculum Dropout

Spatial-temporal (ST) graph modeling, such as traffic speed forecasting and taxi demand prediction, is an important task in deep learning area. However, for the nodes in the graph, their ST patterns can vary greatly in difficulties for modeling, owning to the heterogeneous nature of ST data. We argue that unveiling the nodes to the model in a meaningful order, from easy to complex, can provide performance improvements over traditional training procedure. The idea has its root in Curriculum Learning, which suggests in the early stage of training models can be sensitive to noise and difficult samples. In this paper, we propose ST-Curriculum Dropout, a novel and easy-to-implement strategy for spatial-temporal graph modeling. Specifically, we evaluate the learning difficulty of each node in high-level feature space and drop those difficult ones out to ensure the model only needs to handle fundamental ST relations at the beginning, before gradually moving to hard ones. Our strategy can be applied to any canonical deep learning architecture without extra trainable parameters, and extensive experiments on a wide range of datasets are conducted to illustrate that, by controlling the difficulty level of ST relations as the training progresses, the model is able to capture better representation of the data and thus yields better generalization.

Blood flow characterization in nailfold capillary using optical flow-assisted two-stream network and spatial-temporal image

The blood flow velocity in the nailfold capillary is an important indicator of the status of microcirculation. The conventional manual processing method is both laborious and prone to human artifacts. A feasible way to solve this problem is to use machine learning to assist in image processing and diagnosis. Inspired by the Two-Stream Convolutional Networks, this study proposes an optical flow-assisted two-stream network to segment nailfold blood vessels. Firstly, we use U-Net as the spatial flow network and the dense optical flow as the temporal stream. The results show that the optical flow information can effectively improve the integrity of the segmentation of blood vessels. The overall accuracy is 94.01 %, the Dice score is 0.8099, the IoU score is 0.6806, and the VOE score is 0.3194. Secondly, The flow velocity of the segmented blood vessel is determined by constructing the spatial-temporal (ST) image. The blood flow velocity evaluated is consistent with the typical blood flow speed reported. This study proposes a novel two-stream network for blood vessel segmentation of nailfold capillary images. Combined with ST image and line detection method, it provides an effective workflow for measuring the blood flow velocity of nailfold capillaries.

In vitro to in vivo acetaminophen hepatotoxicity extrapolation using classical schemes, pharmacodynamic models and a multiscale spatial-temporal liver twin.

In vitro to in vivo extrapolation represents a critical challenge in toxicology. In this paper we explore extrapolation strategies for acetaminophen (APAP) based on mechanistic models, comparing classical (CL) homogeneous compartment pharmacodynamic (PD) models and a spatial-temporal (ST), multiscale digital twin model resolving liver microarchitecture at cellular resolution. The models integrate consensus detoxification reactions in each individual hepatocyte. We study the consequences of the two model types on the extrapolation and show in which cases these models perform better than the classical extrapolation strategy that is based either on the maximal drug concentration (Cmax) or the area under the pharmacokinetic curve (AUC) of the drug blood concentration. We find that an CL-model based on a well-mixed blood compartment is sufficient to correctly predict the in vivo toxicity from in vitro data. However, the ST-model that integrates more experimental information requires a change of at least one parameter to obtain the same prediction, indicating that spatial compartmentalization may indeed be an important factor.

A Deep Learning Based Data Recovery Approach for Missing and Erroneous Data of IoT Nodes.

Internet of things (IoT) nodes are deployed in large-scale automated monitoring applications to capture the massive amount of data from various locations in a time-series manner. The captured data are affected due to several factors such as device malfunctioning, unstable communication, environmental factors, synchronization problem, and unreliable nodes, which results in data inconsistency. Data recovery approaches are one of the best solutions to reduce data inconsistency. This research provides a missing data recovery approach based on spatial-temporal (ST) correlation between the IoT nodes in the network. The proposed approach has a clustering phase (CL) and a data recovery (DR) phase. In the CL phase, the nodes can be clustered based on their spatial and temporal relationship, and common neighbors are extracted. In the DR phase, missing data can be recovered with the help of neighbor nodes using the ST-hierarchical long short-term memory (ST-HLSTM) algorithm. The proposed algorithm has been verified on real-world IoT-based hydraulic test rig data sets which are gathered from things speak real-time cloud platform. The algorithm shows approximately 98.5% reliability as compared with the other existing algorithms due to its spatial-temporal features based on deep neural network architecture.

Novel AoD Estimation Algorithms for WSSUS and Non-WSSUS V2V Channel Modeling

In this paper, we propose efficient computational solutions for estimating the statistical properties of wide-sense stationary un-correlative scattering (WSSUS) and non-WSSUS multiple-input multiple-output (MIMO) vehicle-to-vehicle (V2V) channel models. Specifically, in the WSSUS channel models, we first estimate the angle of departure (AoD) for the non-line of sight (NLoS) propagation components. In this manner, the complex channel impulse response (CIR), which are widely used in the existing literature to characterize the wireless channel physical properties, can be estimated on the basis of the estimated AoD and defined model parameters. Conversely, in the non-WSSUS channel models, by estimating the AoD in the initial stage, the real-time complex CIR of the V2V channel model can be estimated on the basis of the estimated AoD and the moving time/velocities/directions of the mobile transmitter (MT) and mobile receiver (MR). In the estimation process of the aforementioned cases, we introduce different solutions to convert the CIRs from the complex domain to the real-value domain, thereby optimizing the computational efficiency for investigating channel characteristics as compared to existing methods. Numerical results of the WSSUS and non-WSSUS MIMO V2V channel characteristics, such as the spatial-temporal (ST) cross-correlation functions (CCFs) and auto-correlation functions (ACFs), are estimated on the basis of the estimated complex CIRs. These results are in agreements with theoretical ones, indicating that the proposed algorithms are practical for estimating the WSSUS and non-WSSUS MIMO V2V channel characteristics.

“When It's Dark in the East, It's Light in the West”: Lifelong Venturing and Accelerated Temporality in Beijing's Urban Villages

Abstract For decades, Chinese rural migrants have been understood as engaging in dagong 打工 (working for a boss), or the selling of their waged labor, conditioned by the global production chain, dormitory regimes, and exploitive labor relations within and beyond factories. Meanwhile, chuangdang 闯荡 (venturing) as a life project that either opposes or extends dagong has been neglected in scholarly studies. Venturing refers to a spatiotemporal condition defined by the uniqueness of urban villages where rural migrants participate in the informal economy on the urban fringes and attach themselves to entrepreneurship and mobility. It is also a multifaceted presentism produced out of the following spatiotemporal conditions: (1) the sense of urgency to get ahead in spatial temporality; (2) an optimistic relationship with unpredictability; and (3) the technologies of mobility in the name of freedom. By highlighting the dilemmas that propel and hinder rural migrants, this article argues that the cultural politics of venturing are a condition that renders rural migrants economically aggressive, yet politically passive. Thereby, it not only prolongs rural migrants’ mobile lives on the urban fringes, but also dampens political actions that claim space through resistance.

Cooperative Distributed Estimation and Control of Multiple Autonomous Vehicles for Range-Based Underwater Target Localization and Pursuit

We study the problem of using single or multiple cooperative autonomous vehicles, called trackers, to localize and pursue an unknown underwater moving target using measurement of the ranges between the tracker(s) and the target. At the motion planning level, each tracker is assigned a spatial–temporal (S-T) curve to track that is generated by the composition of two types of motion: along the target’s trajectory and on a path encircling the target. At the control level, we derive control laws for robust trajectory tracking and show that under mild assumptions on the convergence of the target’s state estimate provided by a suitably designed filter, the tracker converges to and remains in a desired vicinity of the target. For the case of multiple trackers, we propose an efficient distributed estimation and control (DEC) strategy for the trackers that consider explicitly the constraints on the intertracker communication network. To this end, a distributed extended Kalman filter (DEKF) and a distributed control law for cooperative S-T curve tracking are developed to cooperatively pursue and localize the target. Using this setup, all trackers converge to a specified vicinity of the target while keeping an optimal tracker-target relative geometry that maximizes the range information acquired to estimate the target’s state. The stability of the complete closed-loop DEC system is analyzed rigorously and the efficacy of the proposed strategy is illustrated with extensive computer simulations for the 2-D and 3-D cases.

Parallel‐branch network for 3D human pose and shape estimation in video

AbstractHuman pose and shape estimation have developed rapidly, where a skinned multi‐person linear (SMPL) approach performs excellent recently. However, the prior template of the human body in the SMPL model is fixed, thus a deviation may be resulted in the reconstructed body shape if a human body acts sharp movements such as sporting or dancing. To address this problem, we propose a parallel‐branch network including a designed spatial–temporal (ST) branch and a SMPL branch. The ST branch essentially performs the 2D‐to‐3D lifting for more accurate joint prediction, by the designed spatial transformer and temporal transformer. The 3D joints from the ST branch are used to supervise the 3D joints from the SMPL branch and further correct the deviation of the SMPL model. Experiments on some popular benchmarks like 3DPW and MPI‐INF‐3DHP show that our method has better performance than other methods with video input. Our code is available athttps://automation.seu.edu.cn/wcx/list.htm

ASTCN: An Attentive Spatial–Temporal Convolutional Network for Flow Prediction

Flow prediction attracts intensive research interests, since it can offer essential support to many crucial problems in public safety and smart city, e.g., epidemic spread prediction and medical resource allocation optimization. Among all the models in flow prediction, deep learning models (e.g., convolutional neural networks, recurrent neural networks, and graph neural networks) are popular and outperform other statistics and machine learning models, since they can learn intrinsic structures and extract features from spatial–temporal (ST) data. However, most of them set strict temporal periods in the prediction or separate the interaction between spatial and temporal correlations. Therefore, the prediction accuracy is affected. To overcome the difficulties, we propose a flow prediction network attentive spatial–temporal convolutional network (ASTCN), which can effectively handle large-scale flow data and learn complex features. In ASTCN, we leverage an attention mechanism to overcome the previous problem of strict temporal periods, and can effectively fuse ST data with multiple factors from different time-series sources. Furthermore, we propose a causal 3-D convolutional layer based on temporal convolutional networks (TCNs). It can simultaneously extract both spatial and temporal features to improve the prediction accuracy. We comprehensively conducted our experiments based on real-world data sets. Experimental results show that ASTCN outperforms the state-of-the-art methods by at least 3.78% in root mean square error. Therefore, ASTCN is a potential solution to other large-scale ST problems.