Spatial Temporal Graph Convolutional Network Research Articles

Incorporating multimodal context information into traffic speed forecasting through graph deep learning

Accurate traffic speed forecasting is a prerequisite for anticipating future traffic status and increasing the resilience of intelligent transportation systems. However, most studies ignore the involvement of context information ubiquitously distributed over the urban environment to boost speed prediction. The diversity and complexity of context information also hinder incorporating it into traffic forecasting. Therefore, this study proposes a multimodal context-based graph convolutional neural network (MCGCN) model to fuse context data into traffic speed prediction, including spatial and temporal contexts. The proposed model comprises three modules, ie (a) hierarchical spatial embedding to learn spatial representations by organizing spatial contexts from different dimensions, (b) multivariate temporal modeling to learn temporal representations by capturing dependencies of multivariate temporal contexts and (c) attention-based multimodal fusion to integrate traffic speed with the spatial and temporal context representations for multi-step speed prediction. We conduct extensive experiments in Singapore. Compared to the baseline model (spatial-temporal graph convolutional network, STGCN), our results demonstrate the importance of multimodal contexts with the mean-absolute-error improvement of 0.29 km/h, 0.45 km/h and 0.89 km/h in 30-min, 60-min and 120-min speed prediction, respectively. We also explore how different contexts affect traffic speed forecasting, providing references for stakeholders to understand the relationship between context information and transportation systems.

Study on the Interaction Behaviors Identification of Construction Workers Based on ST-GCN and YOLO.

The construction industry is accident-prone, and unsafe behaviors of construction workers have been identified as a leading cause of accidents. One important countermeasure to prevent accidents is monitoring and managing those unsafe behaviors. The most popular way of detecting and identifying workers' unsafe behaviors is the computer vision-based intelligent monitoring system. However, most of the existing research or products focused only on the workers' behaviors (i.e., motions) recognition, limited studies considered the interaction between man-machine, man-material or man-environments. Those interactions are very important for judging whether the workers' behaviors are safe or not, from the standpoint of safety management. This study aims to develop a new method of identifying construction workers' unsafe behaviors, i.e., unsafe interaction between man-machine/material, based on ST-GCN (Spatial Temporal Graph Convolutional Networks) and YOLO (You Only Look Once), which could provide more direct and valuable information for safety management. In this study, two trained YOLO-based models were, respectively, used to detect safety signs in the workplace, and objects that interacted with workers. Then, an ST-GCN model was trained to detect and identify workers' behaviors. Lastly, a decision algorithm was developed considering interactions between man-machine/material, based on YOLO and ST-GCN results. Results show good performance of the developed method, compared to only using ST-GCN, the accuracy was significantly improved from 51.79% to 85.71%, 61.61% to 99.11%, and 58.04% to 100.00%, respectively, in the identification of the following three kinds of behaviors, throwing (throwing hammer, throwing bottle), operating (turning on switch, putting bottle), and crossing (crossing railing and crossing obstacle). The findings of the study have some practical implications for safety management, especially workers' behavior monitoring and management.

Improved spatial-temporal graph convolutional networks for upper limb rehabilitation assessment based on precise posture measurement.

After regular rehabilitation training, paralysis sequelae can be significantly reduced in patients with limb movement disorders caused by stroke. Rehabilitation assessment is the basis for the formulation of rehabilitation training programs and the objective standard for evaluating the effectiveness of training. However, the quantitative rehabilitation assessment is still in the experimental stage and has not been put into clinical practice. In this work, we propose improved spatial-temporal graph convolutional networks based on precise posture measurement for upper limb rehabilitation assessment. Two Azure Kinect are used to enlarge the angle range of the visual field. The rigid body model of the upper limb with multiple degrees of freedom is established. And the inverse kinematics is optimized based on the hybrid particle swarm optimization algorithm. The self-attention mechanism map is calculated to analyze the role of each upper limb joint in rehabilitation assessment, to improve the spatial-temporal graph convolution neural network model. Long short-term memory is built to explore the sequence dependence in spatial-temporal feature vectors. An exercise protocol for detecting the distal reachable workspace and proximal self-care ability of the upper limb is designed, and a virtual environment is built. The experimental results indicate that the proposed posture measurement method can reduce position jumps caused by occlusion, improve measurement accuracy and stability, and increase Signal Noise Ratio. By comparing with other models, our rehabilitation assessment model achieved the lowest mean absolute deviation, root mean square error, and mean absolute percentage error. The proposed method can effectively quantitatively evaluate the upper limb motor function of stroke patients.

Predicting electric vehicle charging demand using a heterogeneous spatio-temporal graph convolutional network

Short-term Electric Vehicle (EV) charging demand prediction is an essential task in the fields of smart grid and intelligent transportation systems, as understanding the spatiotemporal distribution of charging demand over the next few hours could help operators of charging stations and the grid to take measures (e.g., dynamic pricing) in response to varying charging demand. This study proposed a heterogeneous spatial–temporal graph convolutional network to predict the EV charging demand at different spatial and temporal resolutions. Specifically, we first learned the spatial correlations between charging regions by constructing heterogeneous graphs, i.e., a geographic graph and a demand graph. Then, we used graph convolutional layers and gated recurrent units to extract spatio-temporal features in the observations. Further, we designed a region-specific prediction module that grouped regions based on graph embedding and point of interest (POI) data for prediction. We used a large real-world GPS dataset which contained over 76,000 private EVs in Beijing for model training and validation. The results showed that, compared with recently popular spatio-temporal prediction methods, the proposed model had superior prediction accuracy and steady performance at different scales of regions. In addition, we conducted ablation studies and hyperparameter sensitivity tests. The results suggested that incorporating the demand graph and geographic graph could help improve model performance.

Eye-Blink Event Detection Using a Neural-Network-Trained Frame Segment for Woman Drivers in Saudi Arabia

Women have been allowed to drive in Saudi Arabia since 2018, revoking a 30-year ban that also adhered to the traffic rules provided in the country. Conventional drivers are often monitored for safe driving by monitoring their facial reactions, eye blinks, and expressions. As driving experience and vehicle handling features have been less exposed to novice women drivers in Saudi Arabia, technical assistance and physical observations are mandatory. Such observations are sensed as images/video frames for computer-based analyses. Precise computer vision processes are employed for detecting and classifying events using image processing. The identified events are unique to novice women drivers in Saudi Arabia, assisting with their vehicle usage. This article introduces the Event Detection using Segmented Frame (ED-SF) method to improve the abnormal Eye-Blink Detection (EBD) of women drivers. The eye region is segmented using variation pixel extraction in this process. The pixel extraction process requires textural variation identified from different frames. The condition is that the frames are to be continuous in the event detection. This method employs a convolution neural network with two hidden layer processes. In the first layer, continuous and discrete frame differentiations are identified. The second layer is responsible for segmenting the eye region, devouring the textural variation. The variations and discrete frames are used for training the neural network to prevent segment errors in the extraction process. Therefore, the frame segment changes are used for Identifying the expressions through different inputs across different texture luminosities. This method applies to less-experienced and road-safety-knowledge-lacking woman drivers who have initiated their driving journey in Saudi-Arabia-like countries. Thus the proposed method improves the EBD accuracy by 9.5% compared to Hybrid Convolutional Neural Networks (HCNN), Long Short-Term Neural Networks (HCNN + LSTM), Two-Stream Spatial-Temporal Graph Convolutional Networks (2S-STGCN), and the Customized Driving Fatigue Detection Method CDFDM.

A spatial–temporal graph deep learning model for urban flood nowcasting leveraging heterogeneous community features

Flood nowcasting refers to near-future prediction of flood status as an extreme weather event unfolds to enhance situational awareness. The objective of this study was to adopt and test a novel structured deep-learning model for urban flood nowcasting by integrating physics-based and human-sensed features. We present a new computational modeling framework including an attention-based spatial–temporal graph convolution network (ASTGCN) model and different streams of data that are collected in real-time, preprocessed, and fed into the model to consider spatial and temporal information and dependencies that improve flood nowcasting. The novelty of the computational modeling framework is threefold: first, the model is capable of considering spatial and temporal dependencies in inundation propagation thanks to the spatial and temporal graph convolutional modules; second, it enables capturing the influence of heterogeneous temporal data streams that can signal flooding status, including physics-based features (e.g., rainfall intensity and water elevation) and human-sensed data (e.g., residents’ flood reports and fluctuations of human activity) on flood nowcasting. Third, its attention mechanism enables the model to direct its focus to the most influential features that vary dynamically and influence the flood nowcasting. We show the application of the modeling framework in the context of Harris County, Texas, as the study area and 2017 Hurricane Harvey as the flood event. Three categories of features are used for nowcasting the extent of flood inundation in different census tracts: (i) static features that capture spatial characteristics of various locations and influence their flood status similarity, (ii) physics-based dynamic features that capture changes in hydrodynamic variables, and (iii) heterogeneous human-sensed dynamic features that capture various aspects of residents’ activities that can provide information regarding flood status. Results indicate that the ASTGCN model provides superior performance for nowcasting of urban flood inundation at the census-tract level, with precision 0.808 and recall 0.891, which shows the model performs better compared with other state-of-the-art models. Moreover, ASTGCN model performance improves when heterogeneous dynamic features are added into the model that solely relies on physics-based features, which demonstrates the promise of using heterogenous human-sensed data for flood nowcasting. Given the results of the comparisons of the models, the proposed modeling framework has the potential to be more investigated when more data of historical events are available in order to develop a predictive tool to provide community responders with an enhanced prediction of the flood inundation during urban flood.

Auto-STGCN: Autonomous Spatial-Temporal Graph Convolutional Network Search

In recent years, many spatial-temporal graph convolutional network (STGCN) models are proposed to deal with the spatial-temporal network data forecasting problem. These STGCN models have their own advantages, i.e., each of them puts forward many effective operations and achieves good prediction results in the real applications. If users can effectively utilize and combine these excellent operations integrating the advantages of existing models, then they may obtain more effective STGCN models thus create greater value using existing work. However, they fail to do so due to the lack of domain knowledge, and there is lack of automated system to help users to achieve this goal. In this article, we fill this gap and propose Auto-STGCN algorithm, which makes use of existing models to automatically explore high-performance STGCN model for specific scenarios. Specifically, we design Unified-STGCN framework, which summarizes the operations of existing architectures, and use parameters to control the usage and characteristic attributes of each operation, so as to realize the parameterized representation of the STGCN architecture and the reorganization and fusion of advantages. Then, we present Auto-STGCN, an optimization method based on reinforcement learning, to quickly search the parameter search space provided by Unified-STGCN, and generate optimal STGCN models automatically. Extensive experiments on real-world benchmark datasets show that our Auto-STGCN can find STGCN models superior to existing STGCN models used for search space construction, which demonstrates the effectiveness of our proposed method.

Multi-cue temporal modeling for skeleton-based sign language recognition.

Sign languages are visual languages used as the primary communication medium for the Deaf community. The signs comprise manual and non-manual articulators such as hand shapes, upper body movement, and facial expressions. Sign Language Recognition (SLR) aims to learn spatial and temporal representations from the videos of the signs. Most SLR studies focus on manual features often extracted from the shape of the dominant hand or the entire frame. However, facial expressions combined with hand and body gestures may also play a significant role in discriminating the context represented in the sign videos. In this study, we propose an isolated SLR framework based on Spatial-Temporal Graph Convolutional Networks (ST-GCNs) and Multi-Cue Long Short-Term Memorys (MC-LSTMs) to exploit multi-articulatory (e.g., body, hands, and face) information for recognizing sign glosses. We train an ST-GCN model for learning representations from the upper body and hands. Meanwhile, spatial embeddings of hand shape and facial expression cues are extracted from Convolutional Neural Networks (CNNs) pre-trained on large-scale hand and facial expression datasets. Thus, the proposed framework coupling ST-GCNs with MC-LSTMs for multi-articulatory temporal modeling can provide insights into the contribution of each visual Sign Language (SL) cue to recognition performance. To evaluate the proposed framework, we conducted extensive analyzes on two Turkish SL benchmark datasets with different linguistic properties, BosphorusSign22k and AUTSL. While we obtained comparable recognition performance with the skeleton-based state-of-the-art, we observe that incorporating multiple visual SL cues improves the recognition performance, especially in certain sign classes where multi-cue information is vital. The code is available at: https://github.com/ogulcanozdemir/multicue-slr.

Large-Scale Network Imputation and Prediction of Traffic Volume Based on Multi-Source Data Collection System

Although newly developed traffic detectors were actively deployed to improve the accuracy and coverage of collecting city-wise traffic state information, the rapid transition of the traffic management system caused the problems of massive data corruption. For the practical application of recovering the missing values, the deep learning-based imputation technique is used, which relies on prediction performance with the consideration of dynamic spatial and temporal characteristics in the traffic state information. However, the existing method requires an assumption that the given data comprise a complete dataset from a single source based on the experiments evaluated on a small scale or long stream of freeways. In this paper, we propose a multi-variable spatio-temporal learning technique based on multi-source traffic state information, which was realized by adopting Attention-based Spatial–Temporal Graph Convolutional Networks (ASTGCN). The proposed imputation method cooperatively aggregates spatial and temporal correlation from two different types of detectors into an integrated framework, which allows us to predict missing volume regardless of the missing rate. Moreover, the study was conducted on a large-scale network that contains the entire road characteristics. Daejeon city has served as a case study to demonstrate the performance, and the results show that the mean absolute error of the proposed method is under 12 vehicles/5 min. Our work indicates that multi-source traffic state information can be utilized to impute city-wide missing traffic volume.

STJA-GCN: A Multi-Branch Spatial–Temporal Joint Attention Graph Convolutional Network for Abnormal Gait Recognition

Early recognition of abnormal gait enables physicians to determine a prompt rehabilitation plan for patients for the most effective treatment and care. The Kinect depth sensor can easily collect skeleton data describing the position of joints in the human body. However, the default human skeleton model of Kinect includes an excessive number of many joints, which limits the accuracy of the gait recognition methods and increases the computational resources required. In this study, we propose an optimized human skeleton model for the Kinect system and streamline the joints using a center-of-mass calculation. We integrate several techniques to propose an end-to-end, spatial–temporal, joint attention graph convolutional network (STJA-GCN) architecture. We conducted experiments with a fivefold cross-validation on two common datasets of information on abnormal gaits to evaluate the performance of the proposed method. The results show that the STJA-GCN achieved 93.17 and 92.08% accuracy on the two datasets, and compared to the original spatial–temporal graph convolutional network (ST-GCN), the recognition accuracy increases by 9.22 and 20.65%, respectively. Overall, the results demonstrate that the STJA-GCN can accurately recognize abnormal gaits and, thus, can support low-cost rehabilitation assessments at community hospitals or in patients’ homes.

HIT-GCN: Spatial-Temporal Graph Convolutional Network Embedded with Heterogeneous Information of Road Network for Traffic Forecasting

In road networks, attribute information carried by road segment nodes, such as weather and points of interest (POI), exhibit strong heterogeneity and often involve one-to-many or many-to-one relationships. However, research on such heterogeneity in traffic prediction is relatively limited. Our research examines how varying the network propagation pattern based on the degree of node-to-node heterogeneity of information affects the model prediction performance. Specifically, at the node level, we use knowledge embedding to generate knowledge vectors that quantify the heterogeneity among the attribute information of a node. At the road network level, we calculate a homogeneity adjacency matrix that captures both the topological structure of the road network and the similarity of node heterogeneity. This adjacency matrix assigns different weights to neighbors based on their homogeneity, guiding the propagation of graph convolutional networks (GCN). Finally, we separate the representation of propagation into self-representation and neighbor representation to extract multi-attribute information, including self, homogeneity, and heterogeneity. Experiments on real datasets demonstrate that the incorporation of our homogeneity adjacency matrix leads to a significant improvement in the accuracy of short-term and long-term prediction compared with previous work on homogeneous and single-dimensional information. Furthermore, our approach maintains its performance advantage over baseline models under different embedding dimensions and parameter settings.

Human Pose Estimation Based on a Spatial Temporal Graph Convolutional Network

To address the problem of poor detection and under-utilization of the spatial relationship between nodes in human pose estimation, a method based on an improved spatial temporal graph convolutional network (ST-GCN) model is proposed. Firstly, upsampling and segmented random sampling strategies are used to effectively solve the problems of class imbalance and the large sequence length of the dataset. Secondly, an improved detection transformer (DETR) structure is added to effectively suppress the generation of non-maximal suppression (NMS) and anchor points, a multi-head attention (M-ATT) module is introduced into each ST-GCN cell to capture richer feature information, and a residual module is introduced into the 9th ST-GCN cell to avoid possible network degradation in deep networks. In addition, strategies such as warmup, regularization, loss functions, and optimizers are configured to improve the model’s performance. The experimental results show that the average percentage of correct keypoints (PCK) of this method are 93.2% and 92.7% for the FSD and MPII datasets, respectively, which is 1.9% and 1.7% higher than the average PCK of the original ST-GCN method. Moreover, the confusion matrix corresponding to this method also indicated that the model has high recognition accuracy. In addition, comparison experiments with ST-GCN and other methods show that the computation of the model corresponding to this method is about 1.7 GFLOPs and the corresponding MACs are about 6.4 GMACs, which is a good performance.

MTGEA: A Multimodal Two-Stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment

Because of societal changes, human activity recognition, part of home care systems, has become increasingly important. Camera-based recognition is mainstream but has privacy concerns and is less accurate under dim lighting. In contrast, radar sensors do not record sensitive information, avoid the invasion of privacy, and work in poor lighting. However, the collected data are often sparse. To address this issue, we propose a novel Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA), which improves recognition accuracy through accurate skeletal features from Kinect models. We first collected two datasets using the mmWave radar and Kinect v4 sensors. Then, we used zero-padding, Gaussian Noise (GN), and Agglomerative Hierarchical Clustering (AHC) to increase the number of collected point clouds to 25 per frame to match the skeleton data. Second, we used Spatial Temporal Graph Convolutional Network (ST-GCN) architecture to acquire multimodal representations in the spatio-temporal domain focusing on skeletal features. Finally, we implemented an attention mechanism aligning the two multimodal features to capture the correlation between point clouds and skeleton data. The resulting model was evaluated empirically on human activity data and shown to improve human activity recognition with radar data only. All datasets and codes are available in our GitHub.

New machine learning application platform for spatial–temporal thermal error prediction and control with STFGCN for ball screw system

The big data platform, which has a high control accuracy and efficiency, is expected to realize the high-accuracy prediction and real-time control of the thermal error (TE) for the ball screw system. But the TE of a ball screw system has significant dynamic spatial–temporal behaviors. The TE model, which fully captures its dynamic spatial–temporal behaviors, has not been reported so far. Moreover, the embedding of the TE model into the big data platform is extremely difficult, and the deployment of the big data platform is time-consuming and relies on the expert knowledge, and the cloud computing used in the big data platform also faces a limited bandwidth pressure of the industrial Internet, leading to the failure of the real-time control. To overcome the above limitations, a novel machine learning application platform (MLAP) is designed based on the cloud-terminal architecture. Moreover, the spatial–temporal fusion graph convolutional network (STFGCN) is proposed for the first time to fully capture of the dynamic spatial–temporal behaviors, and the proposed STFGCN is embedded into the designed MLAP to expedite the training process of the STFGCN model and reduce the bandwidth pressure on the industrial Internet. The prediction performance and robustness of the proposed STFGCN model are compared with that of the traditional multiple linear regression (MLR), long short-term memory (LSTM), gated recurrent unit (GRU), convolutional neural network-long short-term memory (CNN-LSTM), temporal-graph convolutional network (T-GCN), hypergraph neural network (HGNN), spatial–temporal graph convolutional network (STGCN), and attention-based spatial–temporal graph convolutional network (ASTGCN). The results suggest that the evaluation metrics R2 of the MLR, LSTM, GRU, CNN-LSTM, T-GCN, HGNN, STGCN, ASTGCN, and STFGCN are 0.8039, 0.9500, 0.9441, 0.9414, 0.9571, 0.9512, 0.9689, 0.9712, and 0.9812 to show the prediction performance, respectively. Moreover, the training rate is improved by adding the number of the virtual machine nodes on the MLAP. The reduction rate of the machining error is the ratio of the machining error reduction to the original machining error. The positioning error and its fluctuation range of the ball screw system are reduced effectively, and the machining errors with the implementation of the MLAP are decreased by more than 33% and 72% compared with that with the pitch error control and without the TE control, respectively.

HAR-time: human action recognition with time factor analysis on worker operating time

ABSTRACT This research proposed a data analysis framework applying human action recognition (HAR) with filtering processing to recognize human workers’ actions and further calculate the operating time of each action. When applying HAR to recognize the worker’s operation actions on a manufacturing site, the repeated and reciprocating actions easily cause misrecognition. Without the separation of actions in advance, the evaluation of HAR in real-world manufacturing is different from the evaluation based on the open data in the literature. To resolve the practical issues of applying HAR on a manufacturing site, first, the spatial-temporal skeleton coordinates of human joints were generated as the input of the skeleton-based spatial-temporal graph convolutional neural network (ST-GCN) for training the recognition model. Once the human action is recognized in each frame, the filtering processing considering the reference correction and accumulative moving mode (AMM) was proposed to improve the recognition accuracy. The real-world case study of the forging industry was conducted, and the empirical result shows the proposed framework was able to detect human workers’ repeated actions with 89% accuracy. Also, the time analysis based on HAR-Time recognition can correctly detect the associated time features that have highly correlated to the low quality of the product.

Skeleton-Based Fall Detection with Multiple Inertial Sensors Using Spatial-Temporal Graph Convolutional Networks.

The application of wearable devices for fall detection has been the focus of much research over the past few years. One of the most common problems in established fall detection systems is the large number of false positives in the recognition schemes. In this paper, to make full use of the dependence between human joints and improve the accuracy and reliability of fall detection, a fall-recognition method based on the skeleton and spatial-temporal graph convolutional networks (ST-GCN) was proposed, using the human motion data of body joints acquired by inertial measurement units (IMUs). Firstly, the motion data of five inertial sensors were extracted from the UP-Fall dataset and a human skeleton model for fall detection was established through the natural connection relationship of body joints; after that, the ST-GCN-based fall-detection model was established to extract the motion features of human falls and the activities of daily living (ADLs) at the spatial and temporal scales for fall detection; then, the influence of two hyperparameters and window size on the algorithm performance was discussed; finally, the recognition results of ST-GCN were also compared with those of MLP, CNN, RNN, LSTM, TCN, TST, and MiniRocket. The experimental results showed that the ST-GCN fall-detection model outperformed the other seven algorithms in terms of accuracy, precision, recall, and F1-score. This study provides a new method for IMU-based fall detection, which has the reference significance for improving the accuracy and robustness of fall detection.

LCBP-STGCN: A local cube binary pattern spatial temporal graph convolutional network for micro-expression recognition

Automated micro-expression recognition has become a research highlight in the emotion recognition field. Recent works proposed an LCBP (Local Cube Binary Pattern) method for micro-expression recognition and made full use of spatiotemporal features to represent micro-expressions. Nevertheless, LCBP misses the features while ignoring the underlying discriminative information. In this paper, we present an LCBP-STGCN (Local Cube Binary Pattern Spatial-Temporal Graph Convolutional Network) to resolve the problems of LCBP. A new STGCN with the ability to handle non-Euclidean structure data is proposed to extract high-level features of the micro-expression. STGCN is composed of Spatial Graph Convolutional Network (SGCN) to obtain spatial information and Temporal Convolutional Network (TCN) to capture temporal information of micro-expression. To validly establish the spatiotemporal graph structure of SGCN, we apply ROI (Region of Interest) as node position, LCBP features as node information. By the alternating convolution of SGCN and TCN, high-level spatiotemporal features can be obtained. The extensive experiments on four spontaneous micro-expression datasets of SMIC, CASME I, CASME II, and SAMM demonstrate the proposed LCBP-STGCN can effectively recognize micro-expressions and achieve better performance than some state-of-the-arts.

A New Partitioned Spatial–Temporal Graph Attention Convolution Network for Human Motion Recognition

At present, human action recognition can be used in all walks of life, because the skeleton can transmit intuitive information without being affected by environmental factors. However, it only focuses on local information. In order to solve these problems, we introduce a neural network model for human body recognition in this paper. We propose a model named NEW-STGCN-CA. The model is based on a spatial–temporal graph convolution network (ST-GCN), which contains a new partition strategy and coordination attention (CA) mechanism. By integrating the CA attention mechanism model, we enable the network to focus on input-related information, ignore unnecessary information, and prevent information loss. Second, a new partitioning strategy is proposed for the sampled regions, which is used to enhance the connection between local information and global information. We proved that the Top-1 accuracy of the NEW-STGCN-CA model in the NTU-RGB+D 60 dataset reached 84.86%, which was 1.7% higher than the original model; the accuracy of Top-1 on the Kinetics-Skeleton dataset reached 32.40%, which was 3.17% higher than the original model. The experimental results show that NEW-STGCN-CA can effectively improve the algorithm’s accuracy while also having high robustness and performance.

Skeleton-based action recognition with multi-stream, multi-scale dilated spatial-temporal graph convolution network

Skeleton-based action recognition with multi-stream, multi-scale dilated spatial-temporal graph convolution network

Unsafe Mining Behavior Identification Method Based on an Improved ST-GCN

Aiming to solve the problems of large environmental interference and complex types of personnel behavior that are difficult to identify in the current identification of unsafe behavior in mining areas, an improved spatial temporal graph convolutional network (ST-GCN) for miners’ unsafe behavior identification network in a transportation roadway (NP-AGCN) was proposed. First, the skeleton spatial-temporal map constructed using multi-frame human key points was used for behavior recognition to reduce the interference caused by the complex environment of the coal mine. Second, aiming to solve the problem that the original graph structure cannot learn the association relationship between the non-naturally connected nodes, which leads to the low recognition rate of climbing belts, fighting and other behaviors, the graph structure was reconstructed and the original partitioning strategy was changed to improve the recognition ability of the model for multi-joint interaction behaviors. Finally, in order to alleviate the problem that the graph convolution network has difficulty learning global information due to the small receptive field, multiple self-attention mechanisms were introduced into the graph convolution to improve the recognition ability of the model for unsafe behaviors. In order to verify the detection ability of the model regarding identifying unsafe behaviors of personnel in a coal mine belt area, our model was tested on the public datasets NTU-RGB + D and the self-built datasets of unsafe behaviors in a coal mine belt area. The recognition accuracies of the proposed model in the above datasets were 94.7% and 94.1%, respectively, which were 6.4% and 7.4% higher than the original model, which verified that the proposed model had excellent recognition accuracies.