Spatio-temporal Convolutional Neural Network Research Articles

Touch Gesture and Emotion Recognition Using Decomposed Spatiotemporal Convolutions

Touch is one of the most essential and effective means to convey affective feelings and intentions in human communication. For a social robot, the ability to recognize human touch gestures and emotions could help realize efficient and natural human–robot interaction. To this end, an affective touch gesture dataset involving ten kinds of touch gestures and 12 kinds of discrete emotions was built by using a pressure sensor array, in which the acquired touch gesture samples are three-dimensional (3-D) spatiotemporal signals that include the shape appearance and motion dynamics. Due to the excellent performance of convolutional neural networks (CNNs), spatiotemporal CNNs have been effectively verified by researchers for 3-D signal classification. However, the large number of parameters and the high complexity of training 3-D convolution kernels remain to be solved. In this article, a decomposed spatiotemporal convolution was designed for feature representation from the raw touch gesture samples. Specifically, the 3-D kernel was factorized into three 1-D kernels by tensor decomposition. The proposed convolution has a simpler but deeper architecture than standard 3-D convolution, which improves the nonlinear expression ability of the model. Besides, the computation cost can be reduced without compromising recognition accuracy. Using a user-dependent test mode, the proposed method yields the accuracies of up to 92.41% and 72.47% for touch gesture and emotion recognitions, respectively. Experimental results demonstrate the effectiveness of the proposed method, and at the same time, preliminarily verify the feasibility of robot perceiving human emotions through touch.

Human behaviour analysis based on spatio-temporal dual-stream heterogeneous convolutional neural network

Human behaviour analysis based on spatio-temporal dual-stream heterogeneous convolutional neural network

A Spatio-Temporal Convolutional Neural Network Model for Rice Nutrient Level Diagnosis at Rice Panicle Initiation Stage

A Spatio-Temporal Convolutional Neural Network Model for Rice Nutrient Level Diagnosis at Rice Panicle Initiation Stage

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

An Alzheimer’s Disease Identification and Classification Model Based on the Convolutional Neural Network with Attention Mechanisms

MRI image analysis of brain regions based on deep learning can effectively reduce the workload of doctors in reading films and improve the accuracy of diagnosis. Therefore, deep learning models have great application prospects in the classification and prediction of Alzheimer’s patients and normal people. However, the existing research has ignored the correlation between small abnormalities in local brain regions and changes in brain tissues. To this end, this paper studies an Alzheimer’s disease identification and classification model based on the convolutional neural network (CNN) with attention mechanisms. In this paper, the attention mechanisms were introduced from the regional level and the feature level, and the information of brain MRI images was fused from multiple levels to find out the correlation between the slices in brain MRI images. Then, a spatio-temporal graph CNN with dual attention mechanisms was constructed, which made the network model more attentive to the salient channel features while eliminating the impact of certain noise features. The experimental results verified the effectiveness of the constructed model in identification and classification of Alzheimer’s disease.

Geospatial constrained optimization to simulate and predict spatiotemporal trends of air pollutants

For exposure estimation of air pollutants, data measurement errors and modeling uncertainty may lead to estimation bias and abnormal predictions and relationships between variables. This paper proposes a method of geospatial constrained optimization and deep learning to reliably simulate and predict spatiotemporal trends of air pollutants. In the proposed method, k-nearest neighbors (k-NN) was first used to retrieve the nearest samples to spatialize regular local temporal basis functions at each target location or subregion; then, a convolutional neural network (CNN) was used to extrapolate temporal basis functions for prediction. Domain and empirical knowledge was embedded in extensive constrained optimization to obtain reasonable simulations and predictions. Bootstrapping was used to estimate the uncertainty of constrained optimized values. The method reduced the bias in the point estimates and obtained robust predictions and their uncertainty estimates of spatiotemporal trend for each spatial target location. In the location-based validations of NO2, NOx and PM2.5 in California, even with limited noise input, the proposed method captured the primary spatiotemporal variability (correlation with measured values: 0.75–0.91; explaining 55–84% of the variance). In addition, compared with generalized additive spatiotemporal model, kernel smoother and CNN, the proposed method made one-year reliable spatiotemporal forecasts of weekly averages. The proposed method has important implications for reducing the estimation bias and predicting trends in the air pollutant spatiotemporal fields.

MEGnet: Automatic ICA-based artifact removal for MEG using spatiotemporal convolutional neural networks

Magnetoencephalography (MEG) is a functional neuroimaging tool that records the magnetic fields induced by neuronal activity; however, signal from non-neuronal sources can corrupt the data. Eye-blinks, saccades, and cardiac activity are three of the most common sources of non-neuronal artifacts. They can be measured by affixing eye proximal electrodes, as in electrooculography (EOG), and chest electrodes, as in electrocardiography (ECG), however this complicates imaging setup, decreases patient comfort, and can induce further artifacts from movement. This work proposes an EOG- and ECG-free approach to identify eye-blinks, saccades, and cardiac activity signals for automated artifact suppression.The contribution of this work is three-fold. First, using a data driven, multivariate decomposition approach based on Independent Component Analysis (ICA), a highly accurate artifact classifier is constructed as an amalgam of deep 1-D and 2-D Convolutional Neural Networks (CNNs) to automate the identification and removal of ubiquitous whole brain artifacts including eye-blink, saccade, and cardiac artifacts. The specific architecture of this network is optimized through an unbiased, computer-based hyperparameter random search. Second, visualization methods are applied to the learned abstraction to reveal what features the model uses and to bolster user confidence in the model's training and potential for generalization. Finally, the model is trained and tested on both resting-state and task MEG data from 217 subjects, and achieves a new state-of-the-art in artifact detection accuracy of 98.95% including 96.74% sensitivity and 99.34% specificity on the held out test-set. This work automates MEG processing for both clinical and research use, adapts to the acquired acquisition time, and can obviate the need for EOG or ECG electrodes for artifact detection.

STDnet-ST: Spatio-temporal ConvNet for small object detection

Object detection through convolutional neural networks is reaching unprecedented levels of precision. However, a detailed analysis of the results shows that the accuracy in the detection of small objects is still far from being satisfactory. A recent trend that will likely improve the overall object detection success is to use the spatial information operating alongside temporal video information. This paper introduces STDnet-ST, an end-to-end spatio-temporal convolutional neural network for small object detection in video. We define small as those objects under 16×16 px, where the features become less distinctive. STDnet-ST is an architecture that detects small objects over time and correlates pairs of the top-ranked regions with the highest likelihood of containing those small objects. This permits to link the small objects across the time as tubelets. Furthermore, we propose a procedure to dismiss unprofitable object links in order to provide high quality tubelets, increasing the accuracy. STDnet-ST is evaluated on the publicly accessible USC-GRAD-STDdb, UAVDT and VisDrone2019-VID video datasets, where it achieves state-of-the-art results for small objects.

Tokamak visible image sequence recognition using nonlocal spatio-temporal CNN for attention needed area localization

In this paper, we report a study that was conducted to explore the feasibility of developing a system that classifies the image sequence as ‘disruptive’ shot images or ‘non-disruptive’ shot images. The classifier identifies an image sequence as ‘non-disruptive’ shots or ‘disruptive’ shots using a non-local spatio-temporal 3D convolution neural network (CNN) from image sequences from the Tokamak visible imaging diagnostic system. To analyze the classification result, we localize an area that has contributed to the classification of an image sequence. We use class activation mapping (CAM) for CNN with global average pooling to localize the area. To train this classifier, we created a plasma disruption image sequence dataset using the data acquired from the KSTAR experiment. This classifier recognized disruption image sequences on the test dataset with 91.11% accuracy. Analysis of the CAMs of these image sequences revealed that this classifier recognizes the disruption of the plasma with a relative change in brightness over time in areas other than the plasma area of the image. Through this work, we will be able to develop a system that automatically classifies plasma disruption image sequence after experiments.

STQS: Interpretable multi-modal Spatial-Temporal-seQuential model for automatic Sleep scoring

Sleep scoring is an important step for the detection of sleep disorders and usually performed by visual analysis. Since manual sleep scoring is time consuming, machine-learning based approaches have been proposed. Though efficient, these algorithms are black-box in nature and difficult to interpret by clinicians. In this paper, we propose a deep learning architecture for multi-modal sleep scoring, investigate the model's decision making process, and compare the model's reasoning with the annotation guidelines in the AASM manual. Our architecture, called STQS, uses convolutional neural networks (CNN) to automatically extract spatio-temporal features from 3 modalities (EEG, EOG and EMG), a bidirectional long short-term memory (Bi-LSTM) to extract sequential information, and residual connections to combine spatio-temporal and sequential features. We evaluated our model on two large datasets, obtaining an accuracy of 85% and 77% and a macro F1 score of 79% and 73% on SHHS and an in-house dataset, respectively. We further quantify the contribution of various architectural components and conclude that adding LSTM layers improves performance over a spatio-temporal CNN, while adding residual connections does not. Our interpretability results show that the output of the model is well aligned with AASM guidelines, and therefore, the model's decisions correspond to domain knowledge. We also compare multi-modal models and single-channel models and suggest that future research should focus on improving multi-modal models.

Interpretable Stock Anomaly Detection Based on Spatio-Temporal Relation Networks With Genetic Algorithm

Instability in financial markets represents a considerable risk to investors; examples of instability include a market crash caused by systematic risks and abnormal stock price volatility caused by artificial hype. The early detection of abnormal behavior can help investors adjust their strategy and reduce investment risks. We proposed a spatiotemporal convolutional neural network–based relational network (STCNN-RN) model that can learn the complex correlations between multiple financial time-series data sets, and we used genetic algorithms with a constrained gene to discover the time points for outlier companies by fitting the STCNN-RN model; we used these outlier points to identify abnormal situations. Most research on identifying anomalous patterns has been unable to sufficiently explain the reason for anomalies to investors. We applied an interpretability model to enable investors to understand these anomalous time points in relation to companies and discover the key factors giving rise to the anomalies. The experiment results revealed that the proposed model can be used to model multiple financial time-series data sets and to capture anomalous situations in relevant companies. Because this study explored the discovery of anomaly phenomena in all transaction data and the explanation of these abnormalities, investors can understand a stock market situation holistically.

Spatial Pattern Evaluation of Rural Tourism via the Multifactor-Weighted Neural Network Model in the Big Data Era.

The exploration of the evaluation effect of rural tourism spatial pattern based on the multifactor-weighted neural network model in the era of big data aims to optimize the spatial layout of rural tourist attractions. There are plenty of problems such as improper site selection, layout dispersion, and market competition disorder of rural tourism caused by insufficient consideration of planning and tourist market. Hence, the multifactor model after simple weighting is combined with the neural network to construct a spatiotemporal convolution neural network model based on multifactor weighting here to solve these problems. Moreover, the simulation experiment is conducted on the spatial pattern of rural tourism in the Ningxia Hui Autonomous Region to verify the evaluation performance of the constructed model. The results show that the prediction accuracy of the model is 97.69%, which is at least 2.13% higher than that of the deep learning algorithm used by other scholars. Through the evaluation and analysis of the spatial pattern of rural tourist attractions, the spatial distribution of scenic spots in Ningxia has strong stability from 2009 to 2019. Meanwhile, the number of scenic spots in the seven plates has increased and the time cost of scenic spot accessibility has changed significantly. Besides, the change rate of the one-hour isochronous cycle reaches 41.67%. This indicates that the neural network model has high prediction accuracy in evaluating the spatial pattern of rural tourist attractions, which can provide experimental reference for the digital development of the spatial pattern of rural tourism.

The Research of Lip Reading Based on STCNN and ConvLSTM

Aiming at the problems in temporal model during the research of lip reading, a deep learning model is proposed based on spatiotemporal convolutional neural networks (STCNN) and Convolutional Long Short-Term Memory (ConvLSTM). Firstly, STCNN is used to learn the features of the extracted lip image, and then the learned features are sent to ConvLSTM to process the time series data, which is classified by softmax, and finally the CTC loss function is used to optimize the results. Using GRID data set for training, comparing with experiments, it is found that the recognition accuracy of this model achieves 95.0% at the word level. Experiments show that the model can improve the accuracy of lip reading.

Blind quality assessment of omnidirectional videos using spatio-temporal convolutional neural networks

Panoramic videos are different from 2D videos in the spherical viewing mode. Normally, 2D videos are viewed by consumers on flat screen televisions, while panoramic videos need specific head-mounted displays to create virtual reality experience. Therefore, it is meaningful to study a quality evaluation model suitable for panoramic videos, which will bring a better quality of experience in the field of visual consumption. Unfortunately, to the best of our knowledge, there are few omnidirectional video quality assessments especially in no-reference quality evaluation modes. Besides, the existing 2D quality evaluation methods cannot well evaluate panoramic content quality. Based on above motivation and the vacancy of related research, we propose a blind deep learning-driven quality evaluation framework for panoramic videos to address the correlative issues. From the perspective of the viewport format, we eliminate the sampling distortion of panoramic videos in equirectangular projection (ERP) format via cubemap projection (CMP) format projection. In addition, we use a two-stream convolutional network to fully extract the intra-frame and inter-frame (defined as spatio-temporal) information for more comprehensive modeling of panoramic video features. Experimental results show that our method is superior to the existing full-reference and no-reference methods in solving the task of panoramic video quality assessment.

Enforcing Multilabel Consistency for Automatic Spatio-Temporal Assessment of Shoulder Pain Intensity.

The standard clinical assessment of pain is limited primarily to self-reported pain or clinician impression. While the self-reported measurement of pain is useful, in some circumstances it cannot be obtained. Automatic facial expression analysis has emerged as a potential solution for an objective, reliable, and valid measurement of pain. In this study, we propose a video based approach for the automatic measurement of self-reported pain and the observer pain intensity, respectively. To this end, we explore the added value of three self-reported pain scales, i.e., the Visual Analog Scale (VAS), the Sensory Scale (SEN), and the Affective Motivational Scale (AFF), as well as the Observer Pain Intensity (OPI) rating for a reliable assessment of pain intensity from facial expression. Using a spatio-temporal Convolutional Neural Network - Recurrent Neural Network (CNN-RNN) architecture, we propose to jointly minimize the mean absolute error of pain scores estimation for each of these scales while maximizing the consistency between them. The reliability of the proposed method is evaluated on the benchmark database for pain measurement from videos, namely, the UNBC-McMaster Pain Archive. Our results show that enforcing the consistency between different self-reported pain intensity scores collected using different pain scales enhances the quality of predictions and improve the state of the art in automatic self-reported pain estimation. The obtained results suggest that automatic assessment of self-reported pain intensity from videos is feasible, and could be used as a complementary instrument to unburden caregivers, specially for vulnerable populations that need constant monitoring.

Real-time and accurate abnormal behavior detection in videos

Abnormal crowd behavior detection is a hot research topic in the field of computer vision. In order to solve the problems of high computational cost and the imbalance between positive and negative samples, we propose an efficient algorithm that can detect and locate anomalies in videos. In order to solve the problem of less negative samples, the algorithm uses the spatiotemporal autoencoder to identify and extract the negative samples (contain abnormal behaviors) in the dataset in an unsupervised learning method. On this basis, a spatiotemporal convolutional neural network (CNN) is constructed with simple structure and low computational complexity. The supervised training method is used to train the spatiotemporal CNN with positive and negative samples to generate the detection model. Experiments are conducted on the UCSD and UMN datasets. The experiment results show that the proposed algorithm can detect and locate abnormal behaviors in real time (using only CPU), and the accuracy of the algorithm exceeds those of the existing algorithms at both the pixel level and frame level.

4D spatio-temporal convolutional networks for object position estimation in OCT volumes

Abstract Tracking and localizing objects is a central problem in computer-assisted surgery. Optical coherence tomography (OCT) can be employed as an optical tracking system, due to its high spatial and temporal resolution. Recently, 3D convolutional neural networks (CNNs) have shown promising performance for pose estimation of a marker object using single volumetric OCT images. While this approach relied on spatial information only, OCT allows for a temporal stream of OCT image volumes capturing the motion of an object at high volumes rates. In this work, we systematically extend 3D CNNs to 4D spatio-temporal CNNs to evaluate the impact of additional temporal information for marker object tracking. Across various architectures, our results demonstrate that using a stream of OCT volumes and employing 4D spatio-temporal convolutions leads to a 30% lower mean absolute error compared to single volume processing with 3D CNNs.

RAMST-CNN: A Residual and Multiscale Spatio-Temporal Convolution Neural Network for Personal Identification with EEG

RAMST-CNN: A Residual and Multiscale Spatio-Temporal Convolution Neural Network for Personal Identification with EEG

Multi-Term Attention Networks for Skeleton-Based Action Recognition

The same action takes different time in different cases. This difference will affect the accuracy of action recognition to a certain extent. We propose an end-to-end deep neural network called “Multi-Term Attention Networks” (MTANs), which solves the above problem by extracting temporal features with different time scales. The network consists of a Multi-Term Attention Recurrent Neural Network (MTA-RNN) and a Spatio-Temporal Convolutional Neural Network (ST-CNN). In MTA-RNN, a method for fusing multi-term temporal features are proposed to extract the temporal dependence of different time scales, and the weighted fusion temporal feature is recalibrated by the attention mechanism. Ablation research proves that this network has powerful spatio-temporal dynamic modeling capabilities for actions with different time scales. We perform extensive experiments on four challenging benchmark datasets, including the NTU RGB+D dataset, UT-Kinect dataset, Northwestern-UCLA dataset, and UWA3DII dataset. Our method achieves better results than the state-of-the-art benchmarks, which demonstrates the effectiveness of MTANs.

Spatiotemporal Convolutional Neural Network with Convolutional Block Attention Module for Micro-Expression Recognition

A micro-expression is defined as an uncontrollable muscular movement shown on the face of humans when one is trying to conceal or repress his true emotions. Many researchers have applied the deep learning framework to micro-expression recognition in recent years. However, few have introduced the human visual attention mechanism to micro-expression recognition. In this study, we propose a three-dimensional (3D) spatiotemporal convolutional neural network with the convolutional block attention module (CBAM) for micro-expression recognition. First image sequences were input to a medium-sized convolutional neural network (CNN) to extract visual features. Afterwards, it learned to allocate the feature weights in an adaptive manner with the help of a convolutional block attention module. The method was testified in spontaneous micro-expression databases (Chinese Academy of Sciences Micro-expression II (CASME II), Spontaneous Micro-expression Database (SMIC)). The experimental results show that the 3D CNN with convolutional block attention module outperformed other algorithms in micro-expression recognition.