Extract Spatiotemporal Features Research Articles

Super-Resolution Perception Assisted Spatiotemporal Graph Deep Learning Against False Data Injection Attacks in Smart Grid

Developing the deep learning (DL) technique is a promising way to enhance smart grid (SG) cybersecurity. However, previous DL methods require massive attack samples for cyberattack correlation learning, whilst the real-world SG is incapable of providing such a large dataset. Moreover, existing work commonly focuses on extracting temporal features from power grid data for cyberattack detection, while the spatial features are insufficiently investigated. To address these limitations, a spatiotemporal graph deep learning (STGDL)-based scheme is proposed to detect cyberattacks without requiring attack samples. First, the graph convolution and temporal gated convolution are orchestrated to extract spatiotemporal features jointly. Then, a quantile regression training strategy is adopted to give normally operational bounds of state variables in state estimation (SE). It gets rid of limitations on needing attack samples, and the state bounds can indicate cyberattack anomalies. At last, a super-resolution perception (SRP) network is proposed. The SRP network is able to reconstruct the high-frequent data of estimated states from low-frequent SE results, so as to improve the temporal learning ability in the STGDL model. The feasibility and effectiveness of the proposed scheme are validated by conducting comprehensive and extensive experiments on the IEEE 30-bus and 118-bus benchmarks.

Spatiotemporal informer: A new approach based on spatiotemporal embedding and attention for air quality forecasting

Accurate prediction of air pollution is essential for public health protection. Air quality, however, is difficult to predict due to the complex dynamics, and its accurate forecast still remains a challenge. This study suggests a spatiotemporal Informer model, which uses a new spatiotemporal embedding and spatiotemporal attention, to improve AQI forecast accuracy. In the first phase of the proposed forecast mechanism, the input data is transformed by the spatiotemporal embedding. Next, the spatiotemporal attention is applied to extract spatiotemporal features from the embedded data. The final forecast is obtained based on the attention tensors. In the proposed forecast model, the input is a 3-dimensional data that consists of air quality data (AQI, PM2.5, O3, SO2, NO2, CO) and geographic information, and the output is a multi-positional, multi-temporal data that shows the AQI forecast result of all the monitoring stations in the study area. The proposed forecast model was evaluated by air quality data of 34 monitoring stations in Beijing, China. Experiments showed that the proposed forecast model could provide highly accurate AQI forecast: the average of MAPE values for from 1 h to 20 h ahead forecast was 11.61%, and it was much smaller than other models. Moreover, the proposed model provided a highly accurate and stable forecast even at the extreme points. These results demonstrated that the proposed spatiotemporal embedding and attention techniques could sufficiently capture the spatiotemporal correlation characteristics of air quality data, and that the proposed spatiotemporal Informer could be successfully applied for air quality forecasting.

Spatiotemporal adaptive attention graph convolution network for city-level air quality prediction

Air pollution is a leading cause of human diseases. Accurate air quality predictions are critical to human health. However, it is difficult to extract spatiotemporal features among complex spatiotemporal dependencies effectively. Most existing methods focus on constructing multiple spatial dependencies and ignore the systematic analysis of spatial dependencies. We found that besides spatial proximity stations, functional similarity stations, and temporal pattern similarity stations, the shared spatial dependencies also exist in the complete spatial dependencies. In this paper, we propose a novel deep learning model, the spatiotemporal adaptive attention graph convolution model, for city-level air quality prediction, in which the prediction of future short-term series of PM2.5 readings is preferred. Specifically, we encode multiple spatiotemporal dependencies and construct complete spatiotemporal interactions between stations using station-level attention. Among them, we design a Bi-level sharing strategy to extract shared inter-station relationship features between certain stations efficiently. Then we extract multiple spatiotemporal features with multiple decoders, which it is extracted from the complete spatial dependencies between stations. Finally, we fuse multiple spatiotemporal features with a gating mechanism for multi-step predictions. Our model achieves state-of-the-art experimental results in several real-world datasets.

Shot Boundary Detection with 3D Depthwise Convolutions and Visual Attention.

Shot boundary detection is the process of identifying and locating the boundaries between individual shots in a video sequence. A shot is a continuous sequence of frames that are captured by a single camera, without any cuts or edits. Recent investigations have shown the effectiveness of the use of 3D convolutional networks to solve this task due to its high capacity to extract spatiotemporal features of the video and determine in which frame a transition or shot change occurs. When this task is used as part of a scene segmentation use case with the aim of improving the experience of viewing content from streaming platforms, the speed of segmentation is very important for live and near-live use cases such as start-over. The problem with models based on 3D convolutions is the large number of parameters that they entail. Standard 3D convolutions impose much higher CPU and memory requirements than do the same 2D operations. In this paper, we rely on depthwise separable convolutions to address the problem but with a scheme that significantly reduces the number of parameters. To compensate for the slight loss of performance, we analyze and propose the use of visual self-attention as a mechanism of improvement.

Predicting electric vehicle charging demand using a heterogeneous spatio-temporal graph convolutional network

Short-term Electric Vehicle (EV) charging demand prediction is an essential task in the fields of smart grid and intelligent transportation systems, as understanding the spatiotemporal distribution of charging demand over the next few hours could help operators of charging stations and the grid to take measures (e.g., dynamic pricing) in response to varying charging demand. This study proposed a heterogeneous spatial–temporal graph convolutional network to predict the EV charging demand at different spatial and temporal resolutions. Specifically, we first learned the spatial correlations between charging regions by constructing heterogeneous graphs, i.e., a geographic graph and a demand graph. Then, we used graph convolutional layers and gated recurrent units to extract spatio-temporal features in the observations. Further, we designed a region-specific prediction module that grouped regions based on graph embedding and point of interest (POI) data for prediction. We used a large real-world GPS dataset which contained over 76,000 private EVs in Beijing for model training and validation. The results showed that, compared with recently popular spatio-temporal prediction methods, the proposed model had superior prediction accuracy and steady performance at different scales of regions. In addition, we conducted ablation studies and hyperparameter sensitivity tests. The results suggested that incorporating the demand graph and geographic graph could help improve model performance.

Accurate video saliency prediction via hierarchical fusion and temporal recurrence

With the ability to extract spatiotemporal features, 3D convolutional networks have become the mainstream method for Video Saliency Prediction (VSP). However, these methods cannot make full use of hierarchical spatiotemporal features and also lack focus on past salient features, which hinders further improvements in accuracy. To address these issues, we propose a 3D convolutional Network based on Hierarchical Fusion and Temporal Recurrence (HFTR-Net) for VSP. Specifically, we propose a Bi-directional Temporal–Spatial Feature Pyramid (BiTSFP), which adds the flow of shallow location information based on the previous flow of deep semantic information. Then, different from simple addition and concatenation, we design a Hierarchical Adaptive Fusion (HAF) mechanism that can adaptively learn the fusion weights of adjacent features to integrate them appropriately. Moreover, to utilize previous salient information, a Recall 3D convGRU (R3D GRU) module is integrated into the 3D convolution-based method for the first time. It subtly combines the local feature extraction of the 3D backbone with the long-term relationship modeling of the temporal recurrence mechanism. Experimental results on the three common datasets demonstrate that the HFTR-Net outperforms existing state-of-the-art methods in accuracy.

An SNN-Based Bionic Olfactory Signal Processing Network for Odor Recognition

Feature extraction in electronic nose (E-nose) signals is the first signal processing step in machine olfactory systems. Traditional feature extraction methods are often cumbersome and depend greatly on the experience of the researcher. The failure of feature extraction seriously deteriorates the performance of subsequent pattern recognition algorithms. In addition, inherent sensor signal shifts among sensing systems with identical configurations affect E-nose performance. Therefore, due to the unfavorable performance of empirical feature extraction and poor reproducibility, the odor detection ability of E-nose systems has deteriorated. In this paper, a bionic olfactory signal processing (BOSP) network based on a spiking neural network (SNN) is proposed to address these issues. Inspired by the structure of the olfactory bulb in mammals, the system has two components: a virtual olfactory receptor (VOR) layer and a bionic olfactory bulb (BOB) layer. The VORs convert the sensor response signals into spike sequences through two spike coding strategies. The BOBs, consisting of mitral cells and granule cells, extract spatiotemporal features from the spike sequences. The proposed BOSP network is a feature-level model that automatically extracts useful features from the sensor array outputs without applying traditional and tedious feature extraction steps, which can subsequently cascade various classifiers. The results of extensive experiments demonstrate that the BOSP network not only shows better feature extraction capability than several state-of-the-art methods but also exhibits promising shift suppression ability.

Deep Learning-Aided Modulation Recognition for Non-Orthogonal Signals.

Automatic Modulation Recognition (AMR) can obtain the modulation mode of the received signal for subsequent processing without the assistance of the transmitter. Although the existing AMR methods have been mature for the orthogonal signals, these methods face challenges when deployed in non-orthogonal transmission systems due to the superimposed signals. In this paper, we aim to develop efficient AMR methods for both downlink and uplink non-orthogonal transmission signals using deep learning-based data-driven classification methodology. Specifically, for downlink non-orthogonal signals, we propose a Bi-directional Long Short-Term Memory (BiLSTM)-based AMR method that exploits long-term data dependence to automatically learn irregular signal constellation shapes. Transfer learning is further incorporated to improve recognition accuracy and robustness under varying transmission conditions. For uplink non-orthogonal signals, the combinatorial number of classification types explodes exponentially with the number of signal layers, which becomes the major obstacle to AMR. We develop a spatio-temporal fusion network based on the attention mechanism to efficiently extract spatio-temporal features, and network details are optimized according to the superposition characteristics of non-orthogonal signals. Experiments show that the proposed deep learning-based methods outperform their conventional counterparts in both downlink and uplink non-orthogonal systems. In a typical uplink scenario with three non-orthogonal signal layers, the recognition accuracy can approach 96.6% in the Gaussian channel, which is 19% higher than the vanilla Convolution Neural Network.

Spatial–Temporal Correlation Considering Environmental Factor Fusion for Estimating Gross Primary Productivity in Tibetan Grasslands

Gross primary productivity (GPP) is an important indicator in research on carbon cycling in terrestrial ecosystems. High-accuracy GPP prediction is crucial for ecosystem health and climate change assessments. We developed a site-level GPP prediction method based on the GeoMAN model, which was able to extract spatiotemporal features and fuse external environmental factors to predict GPP on the Tibetan Plateau. We evaluated four models’ behavior—Random Forest (RF), Support Vector Machine (SVM), Deep Belief Network (DBN), and GeoMAN—in predicting GPP at nine flux observation sites on the Tibetan Plateau. The GeoMAN model achieved the best results (R2 = 0.870, RMSE = 0.788 g Cm−2 d−1, MAE = 0.440 g Cm−2 d−1). Distance and vegetation type of the flux sites influenced GPP prediction, with the latter being more significant. The different grassland vegetation types exhibited different sensitivity to environmental factors (Ta, PAR, EVI, NDVI, and LSWI) for GPP prediction. Among them, the site located in the alpine swamp meadow was insensitive to changes in environmental factors; the GPP prediction accuracy of the site located in the alpine meadow steppe decreased significantly with the changes in environmental factors; and the GPP prediction accuracy of the site located in the alpine Kobresia meadow also varied with environmental factor changes, but to a lesser extent than the former. This study provides a good reference that deep learning model is able to achieve good accuracy in GPP simulation when considers spatial, temporal, and environmental factors, and the judgement made by deep learning model conforms to basic knowledge in the relevant field.

Field–Road Operation Classification of Agricultural Machine GNSS Trajectories Using Spatio-Temporal Neural Network

The classification that distinguishes whether machines are driving on roads or working in fields based on their global navigation satellite system (GNSS) trajectories is essential for effective management of cross-regional agricultural machinery services in China. In this paper, a novel field–road classification method utilizing multiple deep neural networks (MultiDNN) is proposed to enhance the accuracy of field and road point classification. The MultiDNN model incorporates a bi-directional long short-term memory network (BiLSTM), a topology adaptive graph convolution network (TAG), and a self-attention network (ATT) to effectively extract spatio-temporal features for field–road classification. The BiLSTM is used to capture temporal relationships along the time axis of a trajectory, providing global contextual information for each point. Then, the TAG network is used to obtain the spatio-temporal relationships between adjacent points in a trajectory, offering local contextual information for each point. Finally, the ATT network assigns varying weights to features to emphasize important characteristics. The performance of the MultiDNN model was evaluated using a wheat harvesting trajectory dataset, and the results showed that it achieved a high degree of accuracy, up to 89.75%, outperforming the best baseline method (GCN) by 2.79%.

Dynamic Reservation of Edge Servers via Deep Reinforcement Learning for Connected Vehicles

Edge computing is promising for connected vehicles. As vehicles move, their resource demands for edge servers vary. Thus, it is necessary to reserve edge servers dynamically to meet variable demands. Existing schemes of edge-server reservation usually rely on statistical information of resource demands to make reservations; they are infeasible for connected vehicles, since such schemes are not adaptive to time-varying demands. To this end, a spatio-temporal reinforcement learning scheme called DeepReserve is developed to learn variable demands and then conduct edge-server reservation. Its design is based on the deep deterministic policy gradient algorithm of deep reinforcement learning (DRL), but is featured with several enhancements. First, the fully-connected neural network in DRL is replaced by a convolutional LSTM (ConvLSTM) network to extract spatio-temporal features of resource demands, which highly improves the prediction accuracy of resource demands. Thus, the actions in DRL (i.e., reservation decisions) can adapt to future demands. Second, an action amender is designed to ensure the actions selected by the neural network follow the spatio-temporal correlation. Finally, a training method called DR-Train is designed to stabilize the training procedure for different traffic patterns. DeepReserve is evaluated through extensive experiments on real-world datasets. Results show that it outperforms state-of-the-art approaches.

Intrusion Detection Method for Internet of Vehicles Based on Parallel Analysis of Spatio-Temporal Features.

The problems with network security that the Internet of Vehicles (IoV) faces are becoming more noticeable as it continues to evolve. Deep learning-based intrusion detection techniques can assist the IoV in preventing network threats. However, previous methods usually employ a single deep learning model to extract temporal or spatial features, or extract spatial features first and then temporal features in a serial manner. These methods usually have the problem of insufficient extraction of spatio-temporal features of the IoV, which affects the performance of intrusion detection and leads to a high false-positive rate. To solve the above problems, this paper proposes an intrusion detection method for IoV based on parallel analysis of spatio-temporal features (PA-STF). First, we built an optimal subset of features based on feature correlations of IoV traffic. Then, we used the temporal convolutional network (TCN) and long short-term memory (LSTM) to extract spatio-temporal features in the IoV traffic in a parallel manner. Finally, we fused the spatio-temporal features extracted in parallel based on the self-attention mechanism and used a multilayer perceptron to detect attacks in the Internet of Vehicles. The experimental results show that the PA-STF method reduces the false-positive rate by 1.95% and 1.57% on the NSL-KDD and UNSW-NB15 datasets, respectively, with the accuracy and F1 score also being superior.

Facial Micro-Expression Recognition Using Double-Stream 3D Convolutional Neural Network with Domain Adaptation.

Humans show micro-expressions (MEs) under some circumstances. MEs are a display of emotions that a human wants to conceal. The recognition of MEs has been applied in various fields. However, automatic ME recognition remains a challenging problem due to two major obstacles. As MEs are typically of short duration and low intensity, it is hard to extract discriminative features from ME videos. Moreover, it is tedious to collect ME data. Existing ME datasets usually contain insufficient video samples. In this paper, we propose a deep learning model, double-stream 3D convolutional neural network (DS-3DCNN), for recognizing MEs captured in video. The recognition framework contains two streams of 3D-CNN. The first extracts spatiotemporal features from the raw ME videos. The second extracts variations of the facial motions within the spatiotemporal domain. To facilitate feature extraction, the subtle motion embedded in a ME is amplified. To address the insufficient ME data, a macro-expression dataset is employed to expand the training sample size. Supervised domain adaptation is adopted in model training in order to bridge the difference between ME and macro-expression datasets. The DS-3DCNN model is evaluated on two publicly available ME datasets. The results show that the model outperforms various state-of-the-art models; in particular, the model outperformed the best model presented in MEGC2019 by more than 6%.

A logistic binary Jaya optimization-based channel selection scheme for motor-imagery classification in brain-computer interface

BCI systems use motor imagery to allow users to control external devices through their brain activity. They extract neural signals from the brain using a large number of EEG channels. However, high-dimensional data from multichannel BCI systems increases the computational burden, leading to slower processing and higher costs. In this study, we proposed a Logistic S-shaped Binary Jaya Optimization Algorithm (LS-BJOA), which combines a logistic map with the Jaya optimization algorithm to alleviate the computational burden caused by many channels. The logistic map introduces stochasticity to better simulate the chaotic behavior of brain signals and improve predictive accuracy. Our method initializes a set of three electrodes as a candidate solution and subsequently determines the most relevant channels iteratively. We used a bi-objective fitness function to evaluate the significance of the selected channels, which involves maximizing classification accuracy and minimizing the length of the channel subset. Initially, a fifth-order bandpass filter with Independent Component Analysis (ICA) was applied to filter MI signals and artifacts reduction, respectively. The Regularized Common Spatial Pattern (RCSP) was used to extract spatiotemporal features from the selected channels. Finally, the three classifiers: (1) Support Vector Machine (SVM), (2) Naïve Bayes (NB), and (3) Linear Discriminant Analysis (LDA) were used to determine maximum classification accuracy. The experiment is validated on three public EEG datasets (BCI Competition IV- 2008 – IIA, BCI Competition IV- dataset 1, BCI competition III – dataset IVa). Our method achieved superior classification accuracy (83.59%, 82.09%, and 89.02% on datasets 1, 2, and 3 respectively) with fewer channels than baseline methods. Additionally, computation time was significantly reduced without compromising accuracy. Topographical mapping revealed frontal lobe involvement in MI tasks during physical activities. Topographical mapping of the selected channels showed that the frontal lobe executes various MI tasks during physical activities.

An improved deep learning architecture for multi-object tracking systems

Robust and reliable 3D multi-object tracking (MOT) is essential for autonomous driving in crowded urban road scenes. In those scenarios, accurate data association between tracked objects and incoming new detections is crucial. This paper presents a tracking system based on the Kalman filter that uses a deep learning approach to the association problem. The proposed architecture consists of three neural networks. First, a convolutional LSTM network extracts spatiotemporal features from a sequence of detections of the same track. Then, a Siamese network calculates the degree of similarity between all tracks and the new detections found at each new frame. Finally, a recurrent LSTM network is used to extract 3D and bounding box information. This model follows the tracking-by-detection paradigm and has been trained with track sequences to be able to handle missed observations and to reduce identity switches. A validation test was carried out on the Argoverse dataset to validate the performance of the proposed system. The developed deep learning approach could improve current multi-object tracking systems based on classic algorithms like the Kalman filter.

A novel unsupervised domain adaptation framework based on graph convolutional network and multi-level feature alignment for inter-subject ECG classification

Electrocardiogram (ECG) is an effective non-invasive tool that can detect arrhythmias. Recently, deep learning (DL) has been widely used in ECG classification algorithms. However, differences between subjects lead to data shifts, hindering the further extension of DL algorithms. To solve this problem, we propose a novel multi-level unsupervised domain adaptation framework (MLUDAF) to diagnose arrhythmias. During feature extraction, we use the atrous spatial pyramid pooling residual (ASPP-R) module to extract spatio-temporal features of the samples. Then the graph convolutional network (GCN) module is used to extract the data structure features. During domain adaptation, we design three alignment mechanisms: domain alignment, semantic alignment, and structure alignment. The three alignment strategies are integrated into a unified deep network to guide the feature extractor to extract domain sharing and distinguishable semantic representations, which can reduce the differences between the source and target domains. Experimental results based on the MIT-BIH database show that the proposed method achieves an overall accuracy of 96.8% for arrhythmia detection. Compared to other methods, the proposed method achieves competitive performance. Cross-domain experiments between databases also demonstrate its strong generalizability. Therefore, the proposed method is promising for application in medical diagnosis systems.

Social Relationship Link Inference Based on Graph Convolutional Networks

In the study of social relationship inference, social relationship link inference aims to infer whether there is a social relationship between users. Most previous works applied the unsupervised graph random walk sampling, which had sampling bias and lost much information. In this paper, a Social Relationship Link Inference Based on Graph Convolutional Networks (SLiGCN) is proposed, which learns the spatiotemporal information of check-ins and the impact of related users’ trajectories. It firstly employs the end-to-end supervised learning and applies a recurrent neural network to extract spatiotemporal sequence features from trajectories. Then the graph convolutional network fuses the features of neighboring nodes and employs the fully connected network to infer social relationship links. Finally, it is evaluated with AUC on three real-world datasets. The experiment results show that, compared with baseline models, it not only avoids hand-crafted feature construction that requires much prior knowledge, but also achieves 10% improvement on average.

Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks

Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.

Two-Level Attention Module Based on Spurious-3D Residual Networks for Human Action Recognition.

In recent years, deep learning techniques have excelled in video action recognition. However, currently commonly used video action recognition models minimize the importance of different video frames and spatial regions within some specific frames when performing action recognition, which makes it difficult for the models to adequately extract spatiotemporal features from the video data. In this paper, an action recognition method based on improved residual convolutional neural networks (CNNs) for video frames and spatial attention modules is proposed to address this problem. The network can guide what and where to emphasize or suppress with essentially little computational cost using the video frame attention module and the spatial attention module. It also employs a two-level attention module to emphasize feature information along the temporal and spatial dimensions, respectively, highlighting the more important frames in the overall video sequence and the more important spatial regions in some specific frames. Specifically, we create the video frame and spatial attention map by successively adding the video frame attention module and the spatial attention module to aggregate the spatial and temporal dimensions of the intermediate feature maps of the CNNs to obtain different feature descriptors, thus directing the network to focus more on important video frames and more contributing spatial regions. The experimental results further show that the network performs well on the UCF-101 and HMDB-51 datasets.

A Strong and Robust Skeleton-Based Gait Recognition Method with Gait Periodicity Priors

Gait recognition aims to identify people by their walking patterns. The normal human walking is a periodic movement. However, the existing gait recognition methods rarely make use of the gait periodicity. In this paper, we propose the gait Periodicity-inspired Temporal feature Pyramid aggregator (PTP) that introduces gait periodicity priors into the gait feature extraction, resulting in a strong and robust skeleton-based gait recognition method called CycleGait. Specially, inspired by gait periodicity, PTP adopts multiple parallel temporal convolution operators with pyramid temporal kernel sizes to extract gait temporal features. Then PTP cooperates with spatial Graph Convolutional Network (GCN) to form the GCN-PTP network. CycleGait uses this network to extract spatio-temporal gait features from the sequence of skeleton coordinates. Besides, to make CycleGait more robust and have better performance, we feed more gait samples with various gait cycles into CycleGait by the plug-and-play Irregular Pace Converter (IPC) that can automatically convert normal pace into irregular and reasonable pace. Extensive experiments conducted on CASIA-B dataset and OG RGB+D dataset show that CycleGait has excellent performance in various complex scenarios, namely, cross-view and cross-walking-condition, and becomes one of the best SOTA methods, which not only outperforms the existing best gait recognition methods by a large margin, but also exhibits a significant level of robustness.