Spatial-temporal Network Research Articles

Spatial Temporal Variation Graph Convolutional Networks (STV-GCN) for Skeleton-Based Emotional Action Recognition

The main core purpose of artificial emotional intelligence is to recognize human emotions. Technologies such as facial, semantic, or brainwave recognition applications have been widely proposed. However, the abovementioned recognition techniques for emotional features require a large number of training samples to obtain high accuracy. Human behaviour pattern can be trained and recognized by the continuous movement of the Spatial Temporal Graph Convolution Network (ST-GCN). However, this technology does not distinguish between the speed of delicate emotions, and the speed of human behaviour and delicate changes of emotions cannot be effectively distinguished. This research paper proposes Spatial Temporal Variation Convolutional Network training for human emotion recognition, using skeleton detection technology to calculate the degree of skeleton point change between consecutive actions and using the nearest neighbour algorithm to classify speed levels and train the ST-GCN recognition model to obtain the emotional state. Application of the speed change recognition ability of the Spatial Temporal Variation Graph Convolution Network (STV-GCN) to artificial emotional intelligence calculation makes it possible to efficiently recognize the delicate actions of happy, sad, fear, and angry in human behaviour. The STV-GCN technology proposed in this paper is compared with ST-GCN and can effectively improve the recognition accuracy by more than 50%.

Multi-Scale Spatial Temporal Graph Neural Network for Skeleton-Based Action Recognition

Graph convolutional networks (GCNs) have achieved remarkable performance on skeleton-based action recognition. Existing GCN-based methods usually apply the fixed graph topology and one fixed temporal convolution kernel to extract the spatial features of joints and temporal features, which is from a single-scale perspective. Actually, human actions are coordinated by various body parts in the spatial domain, and exhibit different characteristics in the temporal domain. Therefore, it is appropriate to model the multi-scale information that can enhance both the explainability and stability, which is ignored in current literatures. To address this issue, we propose a multi-scale spatial-temporal graph neural network (MSTGNN) to discover multi-scale discriminative features from spatial and temporal aspects simultaneously. Our contributions are three-folds: 1) For the spatial domain, inspired by the kinematics of the human action, we develop a three-scale graph data structures in a fine-to-coarse way. A novel hybrid spatial pooling module is then proposed to dynamically exploit the global and comprehensive information step-by-step. 2) For the temporal domain, we design a multi-scale temporal convolution module adaptively fusing the temporal features extracted by different scale convolution kernels. 3) As utilizing one-stream architecture instead of multi-stream architecture, the proposed model can be trained in an end-to-end manner. MSTGNN achieves state-of-the-art performance with less computation complexity. Experimental results conducted on two large datasets (NTU-RGB+D and NTU-RGB+D-120) demonstrate the superiority of MSTGNN.

Multi-Featured Spatial-Temporal and Dynamic Multi-Graph Convolutional Network for Metro Passenger Flow Prediction

Multi-Featured Spatial-Temporal and Dynamic Multi-Graph Convolutional Network for Metro Passenger Flow Prediction

Self-adaptive spatial-temporal network based on heterogeneous data for air quality prediction

With the development of society and the rise of people's environmental awareness, air pollution is receiving increased public attention. Accurate air quality prediction can provide useful information for government decision-making and residents' activities. However, accurately predicting future air quality remains a challenging task because of the complex spatial-temporal dependencies of air quality. Previous studies failed to explicitly model these spatial-temporal dependencies. In this paper, we propose a self-adaptive spatial-temporal network (SA-STNet) to efficiently and effectively capture the spatial-temporal dependencies of air quality. In order to effectively aggregate spatial information, we employ a self-adaptive graph convolution module that can learn the latent spatial correlations of air quality automatically. In the temporal dimension, we utilise three independent components to capture the recent, daily-periodic, and weekly-periodic temporal dependencies of air quality, respectively. In addition, our model exploits rich external complementary information by means of a features extraction component. A parametric-matrix-based fusion architecture is used to combine the outputs of different components into a joint representation which is used for generating the final prediction results. Extensive experiments carried out on real-world datasets demonstrate the outstanding performance of our model compared with baselines and state-of-the-art methods.

Alzheimer's Disease Classification With a Cascade Neural Network

Classification of Alzheimer's Disease (AD) has been becoming a hot issue along with the rapidly increasing number of patients. This task remains tremendously challenging due to the limited data and the difficulties in detecting mild cognitive impairment (MCI). Existing methods use gait [or EEG (electroencephalogram)] data only to tackle this task. Although the gait data acquisition procedure is cheap and simple, the methods relying on gait data often fail to detect the slight difference between MCI and AD. The methods that use EEG data can detect the difference more precisely, but collecting EEG data from both HC (health controls) and patients is very time-consuming. More critically, these methods often convert EEG records into the frequency domain and thus inevitably lose the spatial and temporal information, which is essential to capture the connectivity and synchronization among different brain regions. This paper proposes a cascade neural network with two steps to achieve a faster and more accurate AD classification by exploiting gait and EEG data simultaneously. In the first step, we propose attention-based spatial temporal graph convolutional networks to extract the features from the skeleton sequences (i.e., gait) captured by Kinect (a commonly used sensor) to distinguish between HC and patients. In the second step, we propose spatial temporal convolutional networks to fully exploit the spatial and temporal information of EEG data and classify the patients into MCI or AD eventually. We collect gait and EEG data from 35 cognitively health controls, 35 MCI, and 17 AD patients to evaluate our proposed method. Experimental results show that our method significantly outperforms other AD diagnosis methods (91.07 vs. 68.18%) in the three-way AD classification task (HC, MCI, and AD). Moreover, we empirically found that the lower body and right upper limb are more important for the early diagnosis of AD than other body parts. We believe this interesting finding can be helpful for clinical researches.

Understanding and Modeling Urban Mobility Dynamics via Disentangled Representation Learning

Understanding the underlying patterns of the urban mobility dynamics is essential for both the traffic state estimation and management of urban facilities and services. Due to the coupling relationship of generative factors in spatial-temporal domain, it is challenging to model the citywide traffic dynamics under a structural pattern of critical features such as hours of days, days of weeks and weather conditions. To address this challenge, this article develops a disentangled representation learning framework to learn an interpretable factorized representation of the independent data generative factors. In order to make full use of the knowledge on generative factors, this article proposes spatial-temporal generative adversarial network (ST-GAN) to assign the generative factors of traffic flow to the feature vector in latent space and reconstructs the high-dimensional citywide traffic flow from the given factors. With the help of the disentangled representations, the decomposed feature vector in latent space discloses the relationship between underlying patterns and citywide traffic dynamics. Several comprehensively experiments show that ST-GAN not only effectively improves the prediction accuracy but also promisingly characterize structural properties of the traffic evolution process.

A spatial–temporal network analysis of patent transfers from U.S. universities to firms

Universities play an important role in innovation development and are being recognized as a critical element for the global competitiveness of firms. However, there have been very few large-scale empirical studies using public patent transfer datasets to examine patent transfers from universities to firms. This study proposes a workflow that maps and integrates U.S. Patent and Trademark Office issued patent records with patent assignment datasets to result in the study data covering patents and their transfer transactions from 1990 to 2016. This study focuses on patent transfers from U.S. universities for a spatial–temporal analysis at three levels: institutional, state, and national. In addition, the study identifies a technology-oriented network among universities, firms, and technological areas and supports the notion that patent transfers coincide with the development and change of a local region and are affected and driven by policies, economic development, and cultural factors. This study reveals that the geographical distance of patent transfers has been shortened over time, suggesting more local and regional collaborations among universities and businesses. The results of the study can help identify emerging development fields in a given region, potentially leading to policy applications for research and development, strategic planning, and building effective collaboration networks between universities and businesses.

An Improved Deep Spatial-Temporal Hybrid Model for Bus Speed Prediction

For resolving or alleviating the transportation problems, it is necessary to efficiently manage the public transportation and provide public transport services with high quality and advocate green travel, which rely on accurate traffic data. In order to obtain more accurate bus speed in the future, this paper proposed a novel dynamic hierarchical spatial-temporal network model based on Grey Relation Analysis (EGRA), the convolutional neural network (CNN), and the gated recurrent unit (GRU). The proposed model is named the DHSTN; it exploited EGRA to analyze and choose the suitable candidate line sections with high impacts on the target section and, then, construct a multilayer structure based on the CNN, GRU, and attention mechanism to analyze and capture the spatial and temporal dependency, and finally, the extreme learning machine (ELM) is exploited for the fusion of the long-term and short-term dependency to predict the bus speed variation in the next time interval. Comparative experiments indicate that the DHSTN has better performances, the mean absolute error is around 2.6, and it meets the real requirements.

Graph Attention Spatial-Temporal Network With Collaborative Global-Local Learning for Citywide Mobile Traffic Prediction

With the rapid development of mobile cellular technologies and the increasing popularity of mobile and Internet of Things (IoT) devices, timely mobile traffic forecasting with high accuracy becomes more and more critical for proactive network service provisioning and efficient network resource allocation in smart cities. Traditional traffic forecasting methods mostly rely on time series prediction techniques, which fail to capture the complicated dynamic nature and spatial relations of mobile traffic demand. In this paper, we propose a novel deep learning framework, graph attention spatial-temporal network (GASTN), for accurate citywide mobile traffic forecasting, which can capture not only local geographical dependency but also distant inter-region relationship when considering spatial factor. Specifically, GASTN considers spatial correlation through our constructed spatial relation graph and utilizes structural recurrent neural networks to model the global near-far spatial relationships as well as the temporal dependencies. In the framework of GASTN, two attention mechanisms are designed to integrate different effects in a holistic way. Besides, in order to further enhance the prediction performance, we propose a collaborative global-local learning strategy for the training of GASTN, which takes full advantage of the knowledge from both the global model and local models for individual regions and enhance the effectiveness of our model. Extensive experiments on a large-scale real-world mobile traffic dataset demonstrate that our GASTN model dramatically outperforms the state-of-the-art methods. And it reveals that a significant enhancement in the prediction performance of GASTN can be obtained by leveraging the collaborative global-local learning strategy.

ADST: Forecasting Metro Flow Using Attention-Based Deep Spatial-Temporal Networks with Multi-Task Learning

Passenger flow prediction has drawn increasing attention in the deep learning research field due to its great importance in traffic management and public safety. The major challenge of this essential task lies in multiple spatiotemporal correlations that exhibit complex non-linear correlations. Although both the spatial and temporal perspectives have been considered in modeling, most existing works have ignored complex temporal correlations or underlying spatial similarity. In this paper, we identify the unique spatiotemporal correlation of urban metro flow, and propose an attention-based deep spatiotemporal network with multi-task learning (ADST-Net) at a citywide level to predict the future flow from historical observations. ADST-Net uses three independent channels with the same structure to model the recent, daily-periodic and weekly-periodic complicated spatiotemporal correlations, respectively. Specifically, each channel uses the framework of residual networks, the rectified block and the multi-scale convolutions to mine spatiotemporal correlations. The residual networks can effectively overcome the gradient vanishing problem. The rectified block adopts an attentional mechanism to automatically reweigh measurements at different time intervals, and the multi-scale convolutions are used to extract explicit spatial relationships. ADST-Net also introduces an external embedding mechanism to extract the influence of external factors on flow prediction, such as weather conditions. Furthermore, we enforce multi-task learning to utilize transition passenger flow volume prediction as an auxiliary task during the training process for generalization. Through this model, we can not only capture the steady trend, but also the sudden changes of passenger flow. Extensive experimental results on two real-world traffic flow datasets demonstrate the obvious improvement and superior performance of our proposed algorithm compared with state-of-the-art baselines.

Deep Flexible Structured Spatial–Temporal Model for Taxi Capacity Prediction

The prevalence of taxi-hailing applications has brought great convenience to urban travel. People can not only get a taxi anytime and anywhere, but also make an appointment for taxi in advance. Therefore, people are concerned about the current taxi capacity (i.e. the number of vacant taxis) around them. Meanwhile, they also want to know the future capacity to help them choose the appointment time to avoid congestion and plan their itinerary. However, most of the exiting studies only aim to help taxi companies to schedule traffic resources, and cannot consider future travel plans for users. In this paper, we propose the Deep Flexible Structured Spatial–Temporal Model (DFSSTM) to tackle the task. In order to explore more sufficient temporal relationship of data, DFSSTM models the temporal dynamics as three views: period, trend and closeness. Then the Siamese Spatial–Temporal Network (SSTN) is designed for each view, which introduces the Siamese architecture to capture the spatial–temporal dependencies of inflows and outflows simultaneously. Finally, DFSSTM automatically weights each view and fuses the outputs of the three views to get the final prediction. Experimental results on real-world datasets show that the proposed approach outperforms state-of-the-art methods.

Bidirectional Spatial–Temporal Network for Traffic Prediction with Multisource Data

Urban traffic congestion has an obvious spatial and temporal relationship and is relevant to real traffic conditions. Traffic speed is a significant parameter for reflecting congestion of road networks, which is feasible to predict. Traditional traffic forecasting methods have poor accuracy for complex urban road networks, and do not take into account weather and other multisource data. This paper proposes a convolutional neural network (CNN)-based bidirectional spatial–temporal network (CNN-BDSTN) using traffic speed and weather data by crawling electric map information. In CNN-BDSTN, the spatial dependence of traffic network is captured by CNN to compose the time-series input dataset. Bidirectional long short-term memory (LSTM) is introduced to train the convolutional time-series dataset. Compared with linear regression, autoregressive integrated moving average, extreme gradient boosting, LSTM, and CNN-LSTM, CNN-BDSTN presents its ability of spatial and temporal extension and achieves more accurately predicted results. In this case study, traffic speed data of 155 roads and weather information in Urumqi, Xinjiang, People’s Republic of China, with 1-min interval for 5 months are tested by CNN-BDSTN. The experiment results show that the accuracy of CNN-BDSTN with input of weather information is better than the scenario of no weather information, and the average predicted error is less than 5%.

Skeleton-based action recognition with hierarchical spatial reasoning and temporal stack learning network

Skeleton-based action recognition aims to recognize human actions by exploring the inherent characteristics from the given skeleton sequences and has attracted far more attention due to its great important potentials in practical applications. Previous methods have illustrated that learning discriminative spatial and temporal features from the skeleton sequences is a crucial factor to recognize human actions. Nevertheless, how to model spatio-temporal evolutions is still a challenging problem. In this work, we propose a novel model with hierarchical spatial reasoning and temporal stack learning network (HSR-TSL) to explore the discriminative spatial and temporal features for human action recognition, which consists of a hierarchical spatial reasoning network (HSRN) and a temporal stack learning network (TSLN). Specifically, the HSRN employs a hierarchical residual graph neural network to capture two-level spatial features: intra spatial information of each part and body-level structural information between each part. The TSLN models the detailed temporal dynamics of skeleton sequences by a composition of multiple skip-clip LSTMs. During training, we develop a clip-based incremental loss to effectively optimize the model. We perform extensive experiments on five challenging benchmarks to verify the effectiveness of each component of our model. The comparison results illustrate that our approach significantly boosts the performances for skeleton-based action recognition.

Action recognition on continuous video

Video action recognition has been a challenging task over the years. The challenge herein is not only due to the complication in increasing information in videos but also the requirement of an efficient method to retain information over a longer-term where human action would take to perform. This paper proposes a novel framework, named as long-term video action recognition (LVAR) to perform generic action classification in the continuous video. The idea of LVAR is introducing a partial recurrence connection to propagate information within every layer of a spatial-temporal network, such as the well-known C3D. Empirically, we show that this addition allows the C3D network to access long-term information, and subsequently improves action recognition performance with videos of different length selected from both UCF101 and miniKinetics datasets. Further confirmation of our approach is strengthened with experiments on untrimmed video from the Thumos14 dataset.

Deep spatial-temporal networks for crowd flows prediction by dilated convolutions and region-shifting attention mechanism

Flow prediction at a citywide level is of great significance to traffic management and public safety. Since deep learning has achieved success to deal with complex nonlinear problems, it has drawn increasing attention on making crowd flows prediction through neural networks. Generally, convolutional neural network (CNN) and recurrent neural network (RNN) have been applied to model the spatial-temporal dependency of the city. However, there are still two major challenges in predicting flows. First, it is difficult to train the model with the ability to capture both the nearby and distant spatial dependency by deep local convolutions. Second, daily and weekly patterns in temporal dependency are not strictly periodic for their dynamic temporal shifting in each region. To address these issues, we propose a novel deep learning model which called Local-Dilated Region-Shifting Network (LDRSN). LDRSN combines local convolutions with dilated convolutions to learn the nearby and distant spatial dependency. Furthermore, a new region-level attention mechanism is proposed to model the temporal shifting which varies by region. In the experiments, we compare the proposed method with other state-of-the-art methods in two real-world crowd flows datasets. The Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE) were used as the evaluation indexes. The experiment results show the effectiveness of the proposed model.

Spatial-Temporal Synchronous Graph Convolutional Networks: A New Framework for Spatial-Temporal Network Data Forecasting

Spatial-temporal network data forecasting is of great importance in a huge amount of applications for traffic management and urban planning. However, the underlying complex spatial-temporal correlations and heterogeneities make this problem challenging. Existing methods usually use separate components to capture spatial and temporal correlations and ignore the heterogeneities in spatial-temporal data. In this paper, we propose a novel model, named Spatial-Temporal Synchronous Graph Convolutional Networks (STSGCN), for spatial-temporal network data forecasting. The model is able to effectively capture the complex localized spatial-temporal correlations through an elaborately designed spatial-temporal synchronous modeling mechanism. Meanwhile, multiple modules for different time periods are designed in the model to effectively capture the heterogeneities in localized spatial-temporal graphs. Extensive experiments are conducted on four real-world datasets, which demonstrates that our method achieves the state-of-the-art performance and consistently outperforms other baselines.

Spatial-temporal fractal of urban agglomeration travel demand

In modern society, an urban agglomeration is a highly developed spatial form of integrated cities. Travel activities driven by various travel demands frequently take place within an urban agglomeration. Understanding urban agglomeration travel demand is a basic but significant task. The paper constructs spatial–temporal networks for urban agglomeration travel demand via spatial–temporal decomposition and makes explicit analyses of the attributes of the networks. Taiwan and taxi demand, which are a typical urban agglomeration and representative travel demand, respectively, are taken as the empirical objects of the study. It is found that the degree distributions of the spatial–temporal travel demand networks follow power laws, whose exponents monotonically decrease with the growth of the square cells that are used to divide an urban agglomeration to aggregate travel demand. Nevertheless, the fact of following power laws is not influenced by the spatial–temporal granularity of network construction, indicating a spatial–temporal fractal property of urban agglomeration travel demand. The findings contribute to understanding the nature of travel demand and human mobility in the scale of an urban agglomeration.

Research on the construction of spatial-temporal network of multi-attraction smart scenic area based on information technologies

Information technologies such as mobile Internet, big data, artificial intelligence and remote sensing technology are playing an important role in smart scenic area. This study firstly defines the concept of multi-attraction smart scenic area. Secondly, we describe the construction about spatial-temporal network of multi-attraction smart scenic area, which helps to promote the tourists into orderly movement in multi-attraction smart scenic area and relieve congestion phenomenon of multi-attraction smart scenic area.

Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting

Accurate and real-time traffic passenger flows forecasting at transportation hubs, such as subway/bus stations, is a practical application and of great significance for urban traffic planning, control, guidance, etc. Recently deep learning based methods are promised to learn the spatial-temporal features from high non-linearity and complexity of traffic flows. However, it is still very challenging to handle so much complex factors including the urban transportation network topological structures and the laws of traffic flows with spatial and temporal dependencies. Considering both the static hybrid urban transportation network structures and dynamic spatial-temporal relationships among stations from historical traffic passenger flows, a more effective and fine-grained spatial-temporal features learning framework is necessary. In this paper, we propose a novel spatial-temporal incidence dynamic graph neural networks framework for urban traffic passenger flows prediction. We first model dynamic traffic station relationships over time as spatial-temporal incidence dynamic graph structures based on historically traffic passenger flows. Then we design a novel dynamic graph recurrent convolutional neural network, namely Dynamic-GRCNN, to learn the spatial-temporal features representation for urban transportation network topological structures and transportation hubs. To fully utilize the historical passenger flows, we sample the short-term, medium-term and long-term historical traffic data in training, which can capture the periodicity and trend of the traffic passenger flows at different stations. We conduct extensive experiments on different types of traffic passenger flows datasets including subway, taxi and bus flows in Beijing. The results show that the proposed Dynamic-GRCNN effectively captures comprehensive spatial-temporal correlations significantly and outperforms both traditional and deep learning based urban traffic passenger flows prediction methods.

Global relational reasoning with spatial temporal graph interaction networks for skeleton-based action recognition

With the prevalence of accessible depth sensors, dynamic skeletons have attracted much attention as a robust modality for action recognition. Convolutional neural networks (CNNs) excel at modeling local relations within local receptive fields and are typically inefficient at capturing global relations. In this article, we first view the dynamic skeletons as a spatio-temporal graph (STG) and then learn the localized correlated features that generate the embedded nodes of the STG by message passing. To better extract global relational information, a novel model called spatial–temporal graph interaction networks (STG-INs) is proposed, which perform long-range temporal modeling of human body parts. In this model, human body parts are mapped to an interaction space where graph-based reasoning can be efficiently implemented via a graph convolutional network (GCN). After reasoning, global relation-aware features are distributed back to the embedded nodes of the STG. To evaluate our model, we conduct extensive experiments on three large-scale datasets. The experimental results demonstrate the effectiveness of our proposed model, which achieves the state-of-the-art performance.