Local Spatial-temporal Features Research Articles

Self-Emphasizing Network for Continuous Sign Language Recognition

Hand and face play an important role in expressing sign language. Their features are usually especially leveraged to improve system performance. However, to effectively extract visual representations and capture trajectories for hands and face, previous methods always come at high computations with increased training complexity. They usually employ extra heavy pose-estimation networks to locate human body keypoints or rely on additional pre-extracted heatmaps for supervision. To relieve this problem, we propose a self-emphasizing network (SEN) to emphasize informative spatial regions in a self-motivated way, with few extra computations and without additional expensive supervision. Specifically, SEN first employs a lightweight subnetwork to incorporate local spatial-temporal features to identify informative regions, and then dynamically augment original features via attention maps. It's also observed that not all frames contribute equally to recognition. We present a temporal self-emphasizing module to adaptively emphasize those discriminative frames and suppress redundant ones. A comprehensive comparison with previous methods equipped with hand and face features demonstrates the superiority of our method, even though they always require huge computations and rely on expensive extra supervision. Remarkably, with few extra computations, SEN achieves new state-of-the-art accuracy on four large-scale datasets, PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL. Visualizations verify the effects of SEN on emphasizing informative spatial and temporal features. Code is available at https://github.com/hulianyuyy/SEN_CSLR

Long and short spatial-temporal relations model for automatic generation of Labanotation

As a major research field in computer vision, automatic generation of Labanotation has been attracting the attention of many researchers. These researchers use various methods to engage in dance education and protection, but the existing methods do not consider the spatial-temporal dependence modeling of dance movements. Therefore, they do not represent complex dance movements optimally. We propose an automatic generation network of Labanotation based on the long and short spatial-temporal relations. It includes local spatial-temporal feature extraction network, global spatial-temporal feature extraction network, and local and global feature fusion network. It can align the input and output sequences while modeling the skeletal spatial-temporal relation. The local spatial-temporal features among short interval skeleton sequence relationship are obtained through the multi-scale convolution of time and space. The global spatial-temporal features are learned through the transformer network to obtain the relationship among skeleton sequences with long interval. For the output of the two networks, we use the pyramid squeeze attention network to exchange information in long and short spatial-temporal information, achieving complementarity to improve the accuracy of action recognition. Experimental results show that the proposed method outperforms the state-of-the-art methods on laban16 and laban48, which are common datasets for the research of Labanotation.

STGL-GCN: Spatial–Temporal Mixing of Global and Local Self-Attention Graph Convolutional Networks for Human Action Recognition

Human action recognition methods based on skeleton data have been widely studied owing to their strong robustness to illumination and complex backgrounds. Existing methods have achieved good recognition results; however, they have certain challenges, such as the fixed topological structure of the graph, the omission of nonphysical joint correlation, and the inability to extract local spatial–temporal features. Herein, we propose spatial–temporal mixing of global and local self-attention graph convolutional networks (STGL-GCN) using skeleton data. The global self-attention matrix captures the potential dependencies of nonphysical correlations between joints, and the local self-attention matrix determines the connection strength of the physical edges of joints. The matrices are updated together with the convolution parameters in each network layer as the model is trained for optimal graph structure to achieve accurate action expressions Experiments on the NTU-RGBD dataset demonstrate that our model accurately recognizes actions.

A Hybrid-Convolution Spatial–Temporal Recurrent Network For Traffic Flow Prediction

Abstract Accurate traffic flow prediction is valuable for satisfying citizens’ travel needs and alleviating urban traffic pressure. However, it is highly challenging due to the complexity of the urban geospatial structure and the highly nonlinear temporal and spatial dependence on human mobility. Most existing works proposed to rely on strict periods (e.g. daily and weekly) and separate the extraction of temporal and spatial features. Besides, most Recurrent Neural Network (RNN)-based models either fail to capture variations of spatial–temporal features in adjacent timestamps or ignore details of closeness. In this paper, we propose a Multi-attention based Hybrid-convolution Spatial-temporal Recurrent Network (MHSRN) for region-based traffic flow prediction. In MHSRN, we leverage a hybrid-convolution module to capture both shifting features and rich information at the nearest timestamps, and we apply the downsampling procedure to reduce the computation of RNN-based model. Furthermore, we propose to adopt a space-aware multi-attention module to re-perceive global and local spatial–temporal features. We conduct extensive experiments based on three real-world datasets. The results show that the MHSRN outperforms other challenging baselines by approximately 0.2–8.1% in mean absolute error on all datasets. On datasets other than TaxiBJ, the MHSRN reduces the root mean square error by at least 2.8% compared with the RNN-based model.

CSI-based location-independent Human Activity Recognition with parallel convolutional networks

Human Activity Recognition (HAR) based on Wi-Fi has a broad application prospect in human–computer interaction. Since Wi-Fi signals are sensitive to the environmental changes, the features of the same category of human activity at different locations have significant difference. The existing HAR systems based on Wi-Fi need to re-collect samples or retrain models when recognizing the same activity at new locations, which reduces their practicability in human–computer interaction. To address this challenge, this paper proposes a CSI-based Parallel Convolutional Networks-based location-independent HAR system (CSI-PCNH). CSI-PCNH enhances the inter-class difference by extracting the inter-class features of the different activity samples. In addition, CSI-PCNH improves the generalization ability of activity recognition at any location by extracting the intra-class features of the same category of activity at different locations. In order to obtain the inter-class features and intra-class features of activity samples, we design a parallel convolutional network model which is composed of 3DCNN combined with Channel Attention Mechanism (CAM) and 2DCNN with LSTM to extract the global and local spatial–temporal features of the activity samples. The experimental results show that in the 8 m × 7 m indoor area, the proposed HAR system trained by the activity samples at 12 known locations, the average recognition accuracy for 6 categories of activities at any other 10 locations can reach 91.7%.

DarknetSec: A novel self-attentive deep learning method for darknet traffic classification and application identification

Darknet traffic classification is crucial for identifying anonymous network applications and defensing cyber crimes. Although notable research efforts have been dedicated to classifying darknet traffic by combining machine learning algorithms and elaborately designed features, current methods either heavily depend on hand-crafted features or overlook the global intrinsic relationships among the local features automatically extracted from different data positions, leading to limited classification performance. To tackle this issue, we propose DarknetSec, a novel self-attentive deep learning method for darknet traffic classification and application identification. Concretely, DarknetSec utilizes a cascaded model with a 1-dimensional Convolutional Neural Network (1D CNN) and a bidirectional Long Short-Term Memory (Bi-LSTM) network to capture local spatial-temporal features from the payload content of packets, while the self-attention mechanism is integrated into the abovementioned feature extraction network to mine the intrinsic relationships and hidden connections among the previously extracted content features. In addition, DarknetSec extracts side-channel features from payload statistics to enhance its classification performance. We evaluate DarknetSec on the CICDarknet2020 dataset, which is a representative of darknet traffic covering both Virtual Private Network (VPN) and The Onion Router (Tor) applications. Thorough experiments show that DarknetSec is superior to other state-of-the-art methods, achieving a multiclass accuracy of 92.22% and a macro-F1-score of 92.10%. Additionally, DarknetSec maintains its high accuracy when applied to other encrypted traffic classification tasks.

A Novel Attention-Based Convolution Neural Network for Human Activity Recognition

Human Activity Recognition (HAR) has been widely used for various applications, such as smart homes, healthcare, security and human-robot interaction. In this paper, a novel deep learning model is proposed based on different Convolution Neural Networks (CNN) and the attention mechanism for addressing the HAR problem by using sensory data from accelerometers and gyroscopes and providing high recognition accuracy. The architecture of the proposed model (named LGSTNet) is designed to be able to analyse and extract Local and Global Spatial-Temporal features from sensory data for HAR. The idea of LGSTNet is to segment an activity window into several sub-windows and join the Attention mechanism into an 2D CNN (named Attention-based 2D-CNN module) to precisely learn local spatial-temporal features from those sub-windows. Meanwhile, an 3D CNN (named 3D-CNN module) is designed to learn global spatial-temporal features from the whole window. Both comparison and ablation experiments are conducted on two real-world datasets, named <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">WISDM</monospace> and <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">UCI-HAR</monospace> . By comparing LGSTNet to eight other well-known existing models and its variants on both datasets, LGSTNet achieves the best performance (over 0.8064 and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.9569\,\,{F}_{w}-\textit {Meansure}$ </tex-math></inline-formula> on <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">WISDM</monospace> and <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">UCI-HAR</monospace> , respectively) and is also more robust.

Deeply-learned and spatial–temporal feature engineering for human action understanding

Accurately recognizing various human actions is a key technique in many AI applications, such as visual tracking and human–computer interaction. Aiming at solving the difficulty that the local spatial–temporal features at low-layer is limited and the descriptiveness of middle-level features is weak, we propose a novel human action understanding framework by leveraging the spatial–temporal depth features. More specifically, based on the fact that violent action regions encode highly discriminative information during human action recognition, we employ human depth clue of video images to identify the salient regions for each human. The optical flow characteristics in the regions are utilized as the energy function to measure the regional representativeness. The deep learning architecture is proposed to identify the regions of motions by the energy function. In this way, the sample points are distributed in the areas with intense movements. The collected sample points are utilized as the learned features to capture human action. Based on the learned deep feature, an SVM classifier is used to identify different human action. Comprehensive experimental results have shown that the average recognition accuracy of our human action recognition algorithm reaches 92%, and also exhibits a high robustness to complicated backgrounds.

StNet: Local and Global Spatial-Temporal Modeling for Action Recognition

Despite the success of deep learning for static image understanding, it remains unclear what are the most effective network architectures for spatial-temporal modeling in videos. In this paper, in contrast to the existing CNN+RNN or pure 3D convolution based approaches, we explore a novel spatialtemporal network (StNet) architecture for both local and global modeling in videos. Particularly, StNet stacks N successive video frames into a super-image which has 3N channels and applies 2D convolution on super-images to capture local spatial-temporal relationship. To model global spatialtemporal structure, we apply temporal convolution on the local spatial-temporal feature maps. Specifically, a novel temporal Xception block is proposed in StNet, which employs a separate channel-wise and temporal-wise convolution over the feature sequence of a video. Extensive experiments on the Kinetics dataset demonstrate that our framework outperforms several state-of-the-art approaches in action recognition and can strike a satisfying trade-off between recognition accuracy and model complexity. We further demonstrate the generalization performance of the leaned video representations on the UCF101 dataset.

Content-Attention Representation by Factorized Action-Scene Network for Action Recognition

During action recognition in videos, irrelevant motions in the background can greatly degrade the performance of recognizing specific actions with which we actually concern ourself here. In this paper, a novel deep neural network, called factorized action-scene network (FASNet), is proposed to encode and fuse the most relevant and informative semantic cues for action recognition. Specifically, we decompose the FASNet into two components. One is a newly designed encoding network, named content attention network (CANet), which encodes local spatial–temporal features to learn the action representations with good robustness to the noise of irrelevant motions. The other is a fusion network, which integrates the pretrained CANet to fuse the encoded spatial–temporal features with contextual scene feature extracted from the same video, for learning more descriptive and discriminative action representations. Moreover, different from the existing deep learning based tasks for generic action recognition, which applies softmax loss function as the training guidance, we formulate two loss functions for guiding the proposed model to accomplish more specific action recognition tasks, i.e. , the multilabel correlation loss for multilabel action recognition and the triplet loss for complex event detection. Extensive experiments on the Hollywood2 dataset and the TRECVID MEDTest 14 dataset show that our method achieves superior performance compared with the state-of-the-art methods.

Unsupervised Deep Learning of Mid-Level Video Representation for Action Recognition

Current deep learning methods for action recognition rely heavily on large scale labeled video datasets. Manually annotating video datasets is laborious and may introduce unexpected bias to train complex deep models for learning video representation. In this paper, we propose an unsupervised deep learning method which employs unlabeled local spatial-temporal volumes extracted from action videos to learn midlevel video representation for action recognition. Specifically, our method simultaneously discovers mid-level semantic concepts by discriminative clustering and optimizes local spatial-temporal features by two relatively small and simple deep neural networks. The clustering generates semantic visual concepts that guide the training of the deep networks, and the networks in turn guarantee the robustness of the semantic concepts. Experiments on the HMDB51 and the UCF101 datasets demonstrate the superiority of the proposed method, even over several supervised learning methods.

How do you smile? Towards a comprehensive smile analysis system

To better understand the expression of human smile, there have been considerable studies about automatic smile detection. Despite all the research, few attention is paid to analyze a smile in a comprehensive way. In this paper, a smile analysis system is presented to detailedly measure a person's smile, which consists of three main modules: smile detection, smile intensity estimation and spontaneous versus posed (SVP) smile recognition. Firstly, our recent proposed feature, Self-Similarity of Gradients (GSS), is employed to detect smiling facial images in unconstrained scenarios. Secondly, the smile intensity is estimated in terms of different facial regions rather than merely the mouth region, which is also applied in the temporal phase segmentation of a smile. Finally, in SVP smile recognition module, a discriminative learning model (DLM) is proposed based on a local spatial-temporal feature, which devotes to obtaining most robust and discriminative patterns of interest. The first two modules are the bases of the last, preparing a deeper understanding of a smile. Experiments on benchmark databases are carried out and compared with the state-of-the-art methods respectively, which validate the advantages of our approach of SVP smile recognition. Moreover, a comprehensive analysis of human smile is given for the first time to the best of our knowledge, which could pave the way for computers that better assess the emotional states of their users and provide useful and important information in helping the research of psychology and behavior science.

RGBD Human Action Recognition using Multi-Features Combination and K-Nearest Neighbors Classification

In this paper, we present a novel system to analyze human body motions for action recognition task from two sets of features using RGBD videos. The Bag-of-Features approach is used for recognizing human action by extracting local spatialtemporal features and shape invariant features from all video frames. These feature vectors are computed in four steps: Firstly, detecting all interest keypoints from RGB video frames using Speed-Up Robust Features and filters motion points using Motion History Image and Optical Flow, then aligned these motion points to the depth frame sequences. Secondly, using a Histogram of orientation gradient descriptor for computing the features vector around these points from both RGB and depth channels, then combined these feature values in one RGBD feature vector. Thirdly, computing Hu-Moment shape features from RGBD frames; fourthly, combining the HOG features with Hu-moments features in one feature vector for each video action. Finally, the k-means clustering and the multi-class K-Nearest Neighbor is used for the classification task. This system is invariant to scale, rotation, translation, and illumination. All tested, are utilized on a dataset that is available to the public and used often in the community. By using this new feature combination method improves performance on actions with low movement and reach recognition rates superior to other publications of the dataset.

Evaluation of local spatial–temporal features for cross-view action recognition

Local spatial–temporal feature-based representation is extremely popular for human action recognition. Many spatial–temporal salient point detectors and descriptors have been proposed. Although the promising results have been achieved for action recognition recently, there still exist two severe problems: (1) there is lack of systematic evaluation of local spatial–temporal features on cross-view action recognition; (2) there is lack of a baseline method especially for the task of cross-view action recognition, which can adaptively bridge different feature spaces from multiple views for this cross-domain task. In this paper, we evaluate four popular spatial–temporal features (STIP, Cuboids, MoSIFT, HoG3D) with the framework of transferable dictionary pair learning. This framework can first learn one transferable dictionary pair in both unsupervised and supervised settings. Then, training samples in the source view and testing samples in the target view can be represented with corresponding source and target dictionary respectively to get sparse feature representations, which is used to training classifier for action recognition. In this way, it can map the features from different views into the same feature space to handle the cross-domain task. The evaluation of four spatial–temporal features and the framework of transferable dictionary pair learning are implemented on the popular multi-view human action dataset, IXMAS. The comparative experiments against the representative methods further demonstrate the superiority of this framework on cross-view human action recognition.

Describing Trajectory of Surface Patch for Human Action Recognition on RGB and Depth Videos

This letter proposes a new feature describing the trajectories of surface patches (ToSP) on human bodies for action recognition by a novel scheme of utilizing RGB and depth videos. RGB data contains appearance information by which we track specific patches on body surfaces while depth data contains spatial information by which we describe surface patches. Specifically, we use spatial-temporal interest points as initial points to track in two directions. A ToSP is extracted by keeping the neighborhood in point cloud of each point on the trajectory. By using the temporal pyramid, ToSPs are matched on several levels based on the surface feature extracted from ToSP segments. The proposed feature captures both the shape and position variations of surface patches, thus it has the advantages of trajectories and local spatial-temporal features. The experiment results show that the proposed feature outperforms the existing trajectories based features and depth features.

Fights Behavior Detection Based on Space-Time Interest Points

Human action recognition belongs to the senior visual analysis of computer vision, which involves image processing, artificial intelligence, pattern recognition and so on, is becoming one of the most hot research topic in recent years. In this paper, on the basis of comparative analysis and study towards current methods related to human action recognition, we propose a novel fights behavior detection method which is based on spatial-temporal interest point. Since most information of human action in video are indicated by the space-time interest points of video, we combine spatial-temporal features with motion energy image to describe information of video, and local spatial-temporal features are applied to extract fights behavior model by bags of words. Experimental results show that this method can achieve high accuracy and certain practical value.

A Generalized Pyramid Matching Kernel for Human Action Recognition in Realistic Videos

Human action recognition is an increasingly important research topic in the fields of video sensing, analysis and understanding. Caused by unconstrained sensing conditions, there exist large intra-class variations and inter-class ambiguities in realistic videos, which hinder the improvement of recognition performance for recent vision-based action recognition systems. In this paper, we propose a generalized pyramid matching kernel (GPMK) for recognizing human actions in realistic videos, based on a multi-channel “bag of words” representation constructed from local spatial-temporal features of video clips. As an extension to the spatial-temporal pyramid matching (STPM) kernel, the GPMK leverages heterogeneous visual cues in multiple feature descriptor types and spatial-temporal grid granularity levels, to build a valid similarity metric between two video clips for kernel-based classification. Instead of the predefined and fixed weights used in STPM, we present a simple, yet effective, method to compute adaptive channel weights of GPMK based on the kernel target alignment from training data. It incorporates prior knowledge and the data-driven information of different channels in a principled way. The experimental results on three challenging video datasets (i.e., Hollywood2, Youtube and HMDB51) validate the superiority of our GPMK w.r.t. the traditional STPM kernel for realistic human action recognition and outperform the state-of-the-art results in the literature.

Study on the Generation Model of Weighted Visual Codebook for Action Recognition

The action recognition based on local spatial-temporal feature has attracted more and more attentions. The codebook used in the recognition is usually generated by k-means and the code words are considered to have the same importance. But in fact, it isn’t true. So in this paper, we propose two strategies to measure the importance of different code words and improve the HIK-Kmeans algorithm .The experiment result shows that our improved algorithm increases the accuracy of action recognition.

Sparse representation of local spatial-temporal features with dimensionality reduction for motion recognition

Sparse representation and compressive sensing have attracted substantial interests in computer vision. In this paper, by introducing two new classification criteria, we extended the sparse representation classification method (SRC) for individual images to classify a video that contains a group of local spatial-temporal features. A dictionary is constructed by concatenating all class-specific dictionaries, each of which is learned from a motion class. A test video is assigned to a class label based on the minimum of reconstruction errors of individual local features or overall reconstruction error. Moreover, we compared the effectiveness of the traditional Principal Component Analysis (PCA) and two compressive sensing based dimensionality reduction methods, i.e., Random Matrix projection and Hash Matrix projection in the framework of sparse representation for motion recognition. Experimental results on four public datasets including hand gesture, human facial, human action and mouse behavior demonstrate that the proposed method achieves comparable or higher recognition accuracies compared to other state-of-the-art methods in the literatures. Although the traditional PCA requires more computation to get the transformation matrix, it performs better than the Random Matrix and Hash Matrix projections using gradient features. However, when raw features (i.e., pixel values) are used, the performance of the Random Matrix and Hash Matrix projections is significantly improved.

Human Action Recognition Based on Discriminative Sparse Coding Video Representation

The bag-of-words(BoW) model usually causes large errors and weak discrimination in video representation in video action recognition,and affects the human action recognition accuracy.To solve this problem,a discriminative sparse coding(DSC) video representation algorithm is proposed.It’s a sparse coding framework involving a Fisher discriminative analysis to encode video local spatial-temporal features and increase the video sparse representation discrimination.And an online discriminative dictionary learning algorithm is also proposed to train a dictionary from massive video data.The experiments show that,comparing with the existing algorithms,the proposed algorithm effectively improves human action recognition accuracy.