Spatial-temporal Convolutional Neural Network Research Articles

Temperature prediction of submerged arc furnace in ironmaking industry based on residual spatial-temporal convolutional neural network

The submerged arc furnace is widely regarded as one of the most promising ore smelting technologies. However, the real-time monitoring of the multiple physical fields including electric, thermal, and mas, through computational fluid dynamics demands significant computational resources. This paper introduces the spatial-temporal convolutional neural network algorithm to address this challenge. Initially, the influences of various working conditions on these physical fields are analyzed. Subsequently, a prediction model is developed based on the coupling of these multiple physical fields model. The spatial-temporal convolutional neural network algorithm is then employed to elucidate the main parameter distributions, enabling the automatic real-time detection of temperature variation trends and providing a theoretical foundation for intelligent furnace operation. The findings indicate that the electric field is the predominant factor causing non-uniform heat distribution, with localized overheating primarily occurring at the electrode ends. The application of the proposed model facilitates dynamic prediction of the temperature distribution, establishing relationships between historical and future time steps as well as local and global temperature variations. The reliability of the temperature prediction model is confirmed, with the model achieving an accuracy of 99.76 %, surpassing the 98.18 % accuracy of the traditional multi-layer perceptron model.

DSTCNN: Deformable spatial-temporal convolutional neural network for pedestrian trajectory prediction

Pedestrian trajectory prediction holds significant research value in service robots, autonomous driving, and intelligent monitoring. Currently, most pedestrian trajectory prediction methods focus on data-driven models based on recurrent neural networks, but there is insufficient research on data-driven models based on convolutional neural networks. In this study, we first analyze the two problems in pedestrian trajectory prediction methods based on convolutional neural networks: 1. Previous trajectory prediction methods based on convolutional neural networks have spatial-temporal entanglement problems; 2. These methods are limited by their fixed convolution kernels and cannot accurately model social and temporal interactions. Furthermore, we propose a deformable spatial-temporal convolutional neural network (DSTCNN) to better adapt to the pedestrian trajectory prediction task. The deformable spatial-temporal convolutional neural network models spatial and temporal interactions separately, overcoming the shortcomings of spatial-temporal entanglement. The deformable spatial-temporal convolution also gets rid of the fixed convolution kernel, making the modeling of spatial-temporal interactions more accurate. On the ETH and UCY datasets, the average displacement error and final displacement error of our method are 0.29 and 0.53 meters, respectively. In kernel density estimation, average Mahalanobis distance, and average maximum eigenvalue metrics, our method still achieves better performance compared to baseline methods. Moreover, the deformable spatial-temporal convolutional neural network is a memory-efficient model with only 4.1K parameters.

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

AbstractExposure to varied noisy environments impairs the recognition performance of artificial intelligence‐based speech recognition technologies. Degraded‐performance services can be utilized as limited systems that assure good performance in certain environments, but impair the general quality of speech recognition services. This study introduces an audiovisual speech recognition (AVSR) model robust to various noise settings, mimicking human dialogue recognition elements. The model converts word embeddings and log‐Mel spectrograms into feature vectors for audio recognition. A dense spatial–temporal convolutional neural network model extracts features from log‐Mel spectrograms, transformed for visual‐based recognition. This approach exhibits improved aural and visual recognition capabilities. We assess the signal‐to‐noise ratio in nine synthesized noise environments, with the proposed model exhibiting lower average error rates. The error rate for the AVSR model using a three‐feature multi‐fusion method is 1.711%, compared to the general 3.939% rate. This model is applicable in noise‐affected environments owing to its enhanced stability and recognition rate.

Embedded Spatial–Temporal Convolutional Neural Network Based on Scattered Light Signals for Fire and Interferential Aerosol Classification

Photoelectric smoke detectors are the most cost-effective devices for very early warning fire alarms. However, due to the different light intensity response values of different kinds of fire smoke and interference from interferential aerosols, they have a high false-alarm rate, which limits their popularity in Chinese homes. To address these issues, an embedded spatial–temporal convolutional neural network (EST-CNN) model is proposed for real fire smoke identification and aerosol (fire smoke and interferential aerosols) classification. The EST-CNN consists of three modules, including information fusion, scattering feature extraction, and aerosol classification. Moreover, a two-dimensional spatial–temporal scattering (2D-TS) matrix is designed to fuse the scattered light intensities in different channels and adjacent time slices, which is the output of the information fusion module and the input for the scattering feature extraction module. The EST-CNN is trained and tested with experimental data measured on an established fire test platform using the developed dual-wavelength dual-angle photoelectric smoke detector. The optimal network parameters were selected through extensive experiments, resulting in an average classification accuracy of 98.96% for different aerosols, with only 67 kB network parameters. The experimental results demonstrate the feasibility of installing the designed EST-CNN model directly in existing commercial photoelectric smoke detectors to realize aerosol classification.

A Dense-Sparse Complementary Network for Human Action Recognition based on RGB and Skeleton Modalities

The vulnerability of RGB-based human action recognition in complex environment and variational scenes can be compensated by skeleton modality. Therefore, action recognition methods fusing RGB and skeleton modalities have received increasing attention. However, the recognition performance of the existing methods is still not satisfactory due to the insufficiently optimized sampling, modeling and fusion strategy, even the computational cost is heavy. In this paper, we propose a Dense-Sparse Complementary Network (DSCNet), which aims to leverage the complementary information of the RGB and skeleton modalities at light computational cost to obtain the competitive action recognition performance. Specifically, we first adopt dense and sparse sampling strategies according to the advantages of RGB and skeleton modalities, respectively. And then, we use the skeleton as guiding information to crop the key active region of the persons in the RGB frame, which largely eliminates the interference of the background. Moreover, a Short-Term Motion Extraction Module (STMEM) is proposed to compress the densely sampled RGB frames to fewer frames before feeding them into the backbone network, which avoids a surge in computational cost. And a Sparse Multi-Scale Spatial–Temporal convolutional neural Network (Sparse-MSSTNet) is designed to modeling sparse skeleton. Extensive experiments show that our method effectively combines complementary information of RGB and skeleton modalities to improve recognition accuracy. The DSCNet achieves competitive performance on NTU RGB+D 60, NTU RGB+D 120, PKU-MMD, UAV-human, IKEA ASM and Northwest-UCLA datasets with much less computational cost than exiting methods. The code is available at https://github.com/Maxchengqin/DSCNet.

Dynamic Fall Detection Using Graph-Based Spatial Temporal Convolution and Attention Network

The prevention of falls has become crucial in the modern healthcare domain and in society for improving ageing and supporting the daily activities of older people. Falling is mainly related to age and health problems such as muscle, cardiovascular, and locomotive syndrome weakness, etc. Among elderly people, the number of falls is increasing every year, and they can become life-threatening if detected too late. Most of the time, ageing people consume prescription medication after a fall and, in the Japanese community, the prevention of suicide attempts due to taking an overdose is urgent. Many researchers have been working to develop fall detection systems to observe and notify about falls in real-time using handcrafted features and machine learning approaches. Existing methods may face difficulties in achieving a satisfactory performance, such as limited robustness and generality, high computational complexity, light illuminations, data orientation, and camera view issues. We proposed a graph-based spatial-temporal convolutional and attention neural network (GSTCAN) with an attention model to overcome the current challenges and develop an advanced medical technology system. The spatial-temporal convolutional system has recently proven the power of its efficiency and effectiveness in various fields such as human activity recognition and text recognition tasks. In the procedure, we first calculated the motion along the consecutive frame, then constructed a graph and applied a graph-based spatial and temporal convolutional neural network to extract spatial and temporal contextual relationships among the joints. Then, an attention module selected channel-wise effective features. In the same procedure, we repeat it six times as a GSTCAN and then fed the spatial-temporal features to the network. Finally, we applied a softmax function as a classifier and achieved high accuracies of 99.93%, 99.74%, and 99.12% for ImViA, UR-Fall, and FDD datasets, respectively. The high-performance accuracy with three datasets proved the proposed system’s superiority, efficiency, and generality.

Multi-scale spatial–temporal convolutional neural network for skeleton-based action recognition

Multi-scale spatial–temporal convolutional neural network for skeleton-based action recognition

A mutli-scale spatial-temporal convolutional neural network with contrastive learning for motor imagery EEG classification

Motor imagery (MI) based Brain-computer interfaces (BCIs) have a wide range of applications in the stroke rehabilitation field. However, due to the low signal-to-noise ratio and high cross-subject variation of the electroencephalogram (EEG) signals generated by motor imagery, the classification performance of the existing methods still needs to be improved to meet the need of real practice. To overcome this problem, we propose a multi-scale spatial-temporal convolutional neural network called MSCNet. We introduce the contrastive learning into a multi-temporal convolution scale backbone to further improve the robustness and discrimination of embedding vectors. Experimental results of binary classification show that MSCNet outperforms the state-of-the-art methods, achieving accuracy improvement of 6.04%, 3.98%, and 8.15% on BCIC IV 2a, SMR-BCI, and OpenBMI datasets in subject-dependent manner, respectively. The results show that the contrastive learning method can significantly improve the classification accuracy of motor imagery EEG signals, which provides an important reference for the design of motor imagery classification algorithms.

A spatial temporal graph neural network model for predicting flashover in arbitrary building floorplans

Rapid fire progression, such as flashover, has been one of the leading causes for firefighter deaths and injuries in residential building environments. Due to long computational time of and the required prior knowledge about the fire scene, existing models cannot be used to predict the potential occurrence of flashover in practical firefighting applications. In this paper, a scene-agnostic model (FlashNet) is proposed to predict flashover based on limited heat detector temperature information up to 150 °C. FlashNet utilizes spatial temporal graph convolutional neural networks to effectively learn features from the limited temperature information and to tackle building structure variations. The proposed model is benchmarked against five different state-of-the-art flashover prediction models. Results show that FlashNet outperforms the existing flashover prediction models and it can reliably predict flashover 30 s preceding its occurrence with an overall accuracy of about 92.1%. Ablation study is carried out to examine the effectiveness of different key model components and geometric average adjacency matrix. The research outcomes from this study are expected to enhance firefighters’ situational awareness in the fire scene, protecting them from hazardous fire environments and to pave the way for the development of data-driven prediction systems.

LGST-Drop: label-guided structural dropout for spatial–temporal convolutional neural networks

Region dropout regularization strategies have proven to be highly effective at improving the generalization performance of convolutional neural networks (CNNs) in a variety of computer vision tasks including image classification, object detection, and semantic segmentation because these strategies enable models to focus on a wider range of image region information. However, for action recognition, models need to be able to extract not only useful spatial information but also important temporal and motion information, which cannot be satisfied by traditional regularization strategies. We propose a spatiotemporal dropout strategy to meet the need for regularization in spatial–temporal CNNs. We call it label guided spatial–temporal drop (LGST-Drop); it not only provides effectively structured dropout in the spatial dimension but also regularizes motion information in the temporal dimension. In addition, LGST-Drop’s mask is guided by the predicted categories of the model itself, which we called temporary labels. Extensive experiments on several standard datasets from action recognition domains show the usefulness of the proposed technique in comparison with the previous methods and theirstate-of-the-art variant algorithms.

Multi‐branch angle aware spatial temporal graph convolutional neural network for model‐based gait recognition

AbstractModel‐based gait recognition with skeleton data input has attracted more attention in recent years. The model‐based gait recognition methods take skeletons constructed by body joints as input, which are invariant to changing carrying and clothing conditions. However, previous methods limitedly model the skeleton information in either spatial or temporal domains and ignore the pose variety under different view angles, which results in poor performance for gait recognition. To solve the above problems, we propose the Multi‐Branch Angle Aware Spatial Temporal Graph Convolutional Neural Network to better depict the spatial‐temporal relationship while minimising the interference from the view angles. The model adopts the legacy Spatial Temporal Graph Neural Network (ST‐GCN) as its backbone and relocates it to create independent ST‐GCN branches. The novel Angle Estimator module is designed to predict the skeletons' view angles, which enables the network robust to the changing views. To balance the weights of different body parts and sequence frames, we build a Part‐Frame‐Importance module to redistribute them. Our experiments on the challenging CASIA‐B dataset have proved the efficacy of the proposed method, which achieves state‐of‐the‐art performance under different carrying and clothing conditions.

Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks

BackgroundFreezing of gait (FOG) is a common and debilitating gait impairment in Parkinson’s disease. Further insight into this phenomenon is hampered by the difficulty to objectively assess FOG. To meet this clinical need, this paper proposes an automated motion-capture-based FOG assessment method driven by a novel deep neural network.MethodsAutomated FOG assessment can be formulated as an action segmentation problem, where temporal models are tasked to recognize and temporally localize the FOG segments in untrimmed motion capture trials. This paper takes a closer look at the performance of state-of-the-art action segmentation models when tasked to automatically assess FOG. Furthermore, a novel deep neural network architecture is proposed that aims to better capture the spatial and temporal dependencies than the state-of-the-art baselines. The proposed network, termed multi-stage spatial-temporal graph convolutional network (MS-GCN), combines the spatial-temporal graph convolutional network (ST-GCN) and the multi-stage temporal convolutional network (MS-TCN). The ST-GCN captures the hierarchical spatial-temporal motion among the joints inherent to motion capture, while the multi-stage component reduces over-segmentation errors by refining the predictions over multiple stages. The proposed model was validated on a dataset of fourteen freezers, fourteen non-freezers, and fourteen healthy control subjects.ResultsThe experiments indicate that the proposed model outperforms four state-of-the-art baselines. Moreover, FOG outcomes derived from MS-GCN predictions had an excellent (r = 0.93 [0.87, 0.97]) and moderately strong (r = 0.75 [0.55, 0.87]) linear relationship with FOG outcomes derived from manual annotations.ConclusionsThe proposed MS-GCN may provide an automated and objective alternative to labor-intensive clinician-based FOG assessment. Future work is now possible that aims to assess the generalization of MS-GCN to a larger and more varied verification cohort.

Attention Mechanism Based on Improved Spatial-Temporal Convolutional Neural Networks for Traffic Police Gesture Recognition

Human action recognition has attracted extensive research efforts in recent years, in which traffic police gesture recognition is important for self-driving vehicles. One of the crucial challenges in this task is how to find a representation method based on spatial-temporal features. However, existing methods performed poorly in spatial and temporal information fusion, and how to extract features of traffic police gestures has not been well researched. This paper proposes an attention mechanism based on the improved spatial-temporal convolutional neural network (AMSTCNN) for traffic police gesture recognition. This method focuses on the action part of traffic police and uses the correlation between spatial and temporal features to recognize traffic police gestures, so as to ensure that traffic police gesture information is not lost. Specifically, AMSTCNN integrates spatial and temporal information, uses weight matching to pay more attention to the region where human action occurs, and extracts region proposals of the image. Finally, we use Softmax to classify actions after spatial-temporal feature fusion. AMSTCNN can strongly make use of the spatial-temporal information of videos and select effective features to reduce computation. Experiments on AVA and the Chinese traffic police gesture datasets show that our method is superior to several state-of-the-art methods.

Improved GCN Framework for Human Motion Recognition

Human recognition models based on spatial-temporal graph convolutional neural networks have been gradually developed, and we present an improved spatial-temporal graph convolutional neural network to solve the problems of the high number of parameters and low accuracy of this type of model. The method mainly draws on the inception structure. First, the tensor rotation is added to the graph convolution layer to realize the conversion between graph node dimension and channel dimension and enhance the model’s ability to capture global information for small-scale tasks. Then the inception temporal convolution layer is added to build a multiscale temporal convolution filter to perceive temporal information under different time domains hierarchically from 4-time dimensions. It overcomes the shortcomings of temporal graph convolutional networks in the field of joint relevance of hidden layers and compensates for the information omission of small-scale graph tasks. It also limits the volume of parameters, decreases the arithmetic power, and speeds up the computation. In our experiments, we verify our model on the public dataset NTU RGB + D. Our method reduces the number of the model parameters by 50% and achieves an accuracy of 90% in the CS evaluation system and 94% in the CV evaluation system. The results show that our method not only has high recognition accuracy and good robustness in human behavior recognition applications but also has a small number of model parameters, which can effectively reduce the computational cost.

Motion Action Analysis at Basketball Sports Scene Based on Image Processing

To solve the problems of low accuracy and high time cost in manual recording and statistics of basketball data, an automatic analysis method of motion action under the basketball sports scene based on the spatial temporal graph convolutional neural network is proposed. By using the graph structure in the data structure to model the joints and limbs of the human body, and using the spatial temporal graph structure to model the posture action, the extraction and estimation of human body posture in basketball sports scenes are realized. Then, training combined with transfer learning, the recognition of motion fuzzy posture is realized through the classification and application of a label subset. Finally, using the self-made OpenCV to collect and calibrate NBA basketball videos, the effectiveness of the proposed method is verified by analyzing the motion action. The results show that the proposed method based on the spatial temporal graph convolutional neural network can recognize all kinds of movements in different basketball scenes. The average recognition accuracy is more than 75%. It can be seen that the method has certain practical application value. Compared with the common motion analysis method feature descriptors, the motion action analysis method based on the spatial temporal graph convolution neural network has higher identification accuracy and can be used for motion action analysis in the actual basketball sports scenes.

Cascade Attention-based Spatial-temporal Convolutional Neural Network for Motion Image Posture Recognition

<p>The traditional motion posture recognition methods cannot capture the temporal relationship in a video sequence, which leads to the problem that the recognition effect of time-dependent behaviors is not ideal. Therefore, this paper proposes a cascade attention-based spatial-temporal convolutional neural network for motion posture recognition. Firstly, the convolutional neural network is used to model the time sequence relationship in the video, so as to capture the spatial-temporal information in the video efficiently. At the same time, the cascade attention mechanism is used to improve the low learning ability of spatial features caused by channel information moving on the time axis. Meanwhile, a new spatial-temporal network structure is constructed, which includes the spatial-temporal appearance information flow and spatial-temporal motion information flow. Finally, the weighted average method is used to fuse the two spatial-temporal networks to obtain the final recognition result. Experiments are conducted on UCF101 and HMDB51 datasets, respectively, and the recognition accuracy is 96.8% and 79.6%. Experiment results show that compared with the state-of-the-art network methods, the recognition accuracy with the proposed method has better effect and robustness.</p> <p> </p>

Recognize highly similar sewing gestures by the robot

The autonomous and efficient learning of sewing gestures by robots will bring great convenience to the garment industry. To improve the accuracy of robots in detecting sewing gestures with high similarity, three detection models based on deep learning are proposed in the paper. First, in order to improve the detection accuracy and detection speed of sewing gestures under complex backgrounds, we added a dense connection layer to the low-resolution network layer of YOLO-V3 to enhance the transmission and reuse rate of image features. Secondly, a deeper ResNet50 residual network is introduced to replace the VGG16 basic network in the original SSD model. The feature pyramid structure is used to fuse high-level semantic features and low-level semantic features, which can improve the detection accuracy of small-sized sewing gestures. Finally, the parallel spatial-temporal dual-stream network separately extracts the temporal feature and the spatial feature of sewing gestures. The fusion of time feature and space feature improves the detection accuracy of the coherent sewing gesture. The results show that the suggested three models can effectively detect four sewing gestures with high similarity. Among them, the spatial-temporal two-stream convolutional neural network has the highest detection accuracy. The improved SSD model has faster detection speed than the improved YOLO-V3 model and other mainstream algorithms.

Video-based Fire Smoke Detection Using Temporal-spatial Saliency Features

In this paper, we propose a video based spatial-temporal convolutional neural network for fire smoke recognition. The model concatenates the appearance features and the motion features followed by a convolution layer to implement spatial-temporal feature fusion. To reduce the influence of background of no-smoke, we use an attention module to capture salience features from the input image. Experiments on the self-created dataset show that the presented method is valid, which achieves a detection rate of 97.5% and accuracy rate of 96.8%.

Multi-scale spatialtemporal information deep fusion network with temporal pyramid mechanism for video action recognition

In the deep learning-based video action recognitio, the function of the neural network is to acquire spatial information, motion information, and the associated information of the above two kinds of information over an uneven time span. This paper puts forward a network extracting video sequence semantic information based on deep integration of local Spatial-Temporal information. The network uses 2D Convolutional Neural Network (2DCNN) and Multi Spatial-Temporal scale 3D Convolutional Neural Network (MST_3DCNN) respectively to extract spatial information and motion information. Spatial information and motion information of the same time quantum receive 3D convolutional integration to generate the temporary Spatial-Temporal information of a certain moment. Then, the Spatial-Temporal information of multiple single moments enters Temporal Pyramid Net (TPN) to generate the local Spatial-Temporal information of multiple time scales. Finally, bidirectional recurrent neutral network is used to act on the Spatial-Temporal information of all parts so as to acquire the context information spanning the length of the entire video, which endows the network with video context information extraction capability. Through the experiments on the three video action recognitio common experimental data sets UCF101, UCF11, UCFSports, the Spatial-Temporal information deep fusion network proposed in this paper has a high correct recognition rate in the task of video action recognitio.

Video-based driver action recognition via hybrid spatial–temporal deep learning framework

Driver action recognition aims to distinguish normal driver action and some abnormal driver actions such as leaving the wheel, talking on the phone, diving with smoking, etc. For the purpose of traffic safety, studies on the computer vision technologies for driver action recognition have become especially meaningful. However, this issue is far from being solved, mainly due to the subtle variations between different driver action classes. In this paper, we present a new video-based driver action recognition approach based on the hybrid spatial–temporal deep learning framework. Specifically, we first design an encoder–decoder spatial–temporal convolutional neural network (EDSTCNN) to capture short-term spatial–temporal representation of driver actions jointly with optical flow prediction. Second, we exploit the feature refinement network (FRN) to refine the short-term driver action feature. Then, convolutional long short-term memory network (ConvLSTM) is employed for long-term spatial–temporal fusion. Finally, the fully connected neural network (FCNN) is used for final driver action recognition. In our experiment, we validate the performance of the proposed framework on our self-created datasets, including a simulated driving dataset and a real driving dataset. Extensive experimental results illustrate that the proposed hybrid spatial–temporal deep learning framework obtains the highest accuracy in multiple driver action recognition datasets (98.9% on SEU-DAR-V1 dataset and 97.0% on SEU-DAR-V2 dataset).