Spatio-temporal Feature Representation Research Articles

Squeeze-and-excitation 3D convolutional attention recurrent network for end-to-end speech emotion recognition

Speech emotion recognition (SER) is difficult since emotions are complex and dynamic processes involving multiple dimensions and sub-dimensions. Feature extraction is a challenging step in SER, where relevant features are extracted from the speech to identify emotional states accurately. Overcoming these challenges is essential to ensuring the effectiveness and robustness of the SER system. 3D-convolutional neural networks (3D-CNNs) are successfully used for feature extraction in SER. Speech signals can be converted into spectrogram-like representations where one axis represents time, another frequency, and the third can represent additional context or features. For modeling the dynamic nature of speech, a squeeze-and-excitation-based 3D-CNN model is employed for capturing temporal and spatial features of speech. Attention Gated Recurrent Units (AGRU) are applied to the extracted features for learning long-range temporal dependencies and selecting more informative features. The extraction and selection of the spatio-temporal feature representation lead to a hierarchical representation of the input speech. The fusion of squeeze-and-excitation 3D-CNN and AGRU (named SE3D-CARN) is evaluated on two datasets, EMO-BD and IEMOCAP, to identify various emotional states in speech. The proposed SER model reached an accuracy of 94.2% and 81.1% on the EMO-BD and IEMOCAP datasets, respectively.

Spatiotemporal feature learning for no-reference gaming content video quality assessment

Recently, over-the-top live gaming content video (GCV) services have significantly contributed to the overall internet traffic. Consequently, there is a growing demand of GCV quality assessment (GCVQA) to maintain service quality. Although recent literature has proposed a few GCVQA methods, these mainly focus on extracting spatial features and temporal fusion separately, limiting their performance due to the neglect of spatiotemporal feature learning, which is crucial for GCV as it typically shares spatial and temporal features across frames. To address this, we propose a novel GCVQA model, focusing on GCV spatiotemporal feature learning. First, we employ a multi-task self-supervised learning spatiotemporal pyramid convolutional neural network (STP-CNN) model to extract short-term spatiotemporal quality feature representations (STQFR) of GCVs. Our STP-CNN model specifically extracts multiscale spatiotemporal features from various temporal scales of multi-frames in pyramid mode, enabling dynamic learning of diverse spatiotemporal cues. Subsequently, we propose the differential Transformer model to process all short-term STQFR within a GCV, extracting global spatiotemporal features of GCV to assess the overall quality of GCV. To evaluate the effectiveness of our proposed method, we conducted experiments using four GCVQA datasets. The results demonstrate that our method outperforms existing approaches in predicting the perceived quality of GCV.

JUST-Net: Jointly unrolled cross-domain optimization based spatio-temporal reconstruction network for accelerated 3D myelin water imaging.

We introduced a novel reconstruction network, jointly unrolled cross-domain optimization-based spatio-temporal reconstruction network (JUST-Net), aimed at accelerating 3D multi-echo gradient-echo (mGRE) data acquisition and improving the quality of resulting myelin water imaging (MWI) maps. An unrolled cross-domain spatio-temporal reconstruction network was designed. The main idea is to combine frequency and spatio-temporal image feature representations and to sequentially implement convolution layers in both domains. The k-space subnetwork utilizes shared information from adjacent frames, whereas the image subnetwork applies separate convolutions in both spatial and temporal dimensions. The proposed reconstruction network was evaluated for both retrospectively and prospectively accelerated acquisition. Furthermore, it was assessed in simulation studies and real-world cases with k-space corruptions to evaluate its potential for motion artifact reduction. The proposed JUST-Net enabled highly reproducible and accelerated 3D mGRE acquisition for whole-brain MWI, reducing the acquisition time from fully sampled 15:23 to 2:22 min within a 3-min reconstruction time. The normalized root mean squared error of the reconstructed mGRE images increased by less than 4.0%, and the correlation coefficients for MWI showed a value of over 0.68 when compared to the fully sampled reference. Additionally, the proposed method demonstrated a mitigating effect on both simulated and clinical motion-corrupted cases. The proposed JUST-Net has demonstrated the capability to achieve high acceleration factors for 3D mGRE-based MWI, which is expected to facilitate widespread clinical applications of MWI.

An Effective Video Transformer With Synchronized Spatiotemporal and Spatial Self-Attention for Action Recognition.

Convolutional neural networks (CNNs) have come to dominate vision-based deep neural network structures in both image and video models over the past decade. However, convolution-free vision Transformers (ViTs) have recently outperformed CNN-based models in image recognition. Despite this progress, building and designing video Transformers have not yet obtained the same attention in research as image-based Transformers. While there have been attempts to build video Transformers by adapting image-based Transformers for video understanding, these Transformers still lack efficiency due to the large gap between CNN-based models and Transformers regarding the number of parameters and the training settings. In this work, we propose three techniques to improve video understanding with video Transformers. First, to derive better spatiotemporal feature representation, we propose a new spatiotemporal attention scheme, termed synchronized spatiotemporal and spatial attention (SSTSA), which derives the spatiotemporal features with temporal and spatial multiheaded self-attention (MSA) modules. It also preserves the best spatial attention by another spatial self-attention module in parallel, thereby resulting in an effective Transformer encoder. Second, a motion spotlighting module is proposed to embed the short-term motion of the consecutive input frames to the regular RGB input, which is then processed with a single-stream video Transformer. Third, a simple intraclass frame interlacing method of the input clips is proposed that serves as an effective video augmentation method. Finally, our proposed techniques have been evaluated and validated with a set of extensive experiments in this study. Our video Transformer outperforms its previous counterparts on two well-known datasets, Kinetics400 and Something-Something-v2.

Hybrid-FHR: a multi-modal AI approach for automated fetal acidosis diagnosis

BackgroundIn clinical medicine, fetal heart rate (FHR) monitoring using cardiotocography (CTG) is one of the most commonly used methods for assessing fetal acidosis. However, as the visual interpretation of CTG depends on the subjective judgment of the clinician, this has led to high inter-observer and intra-observer variability, making it necessary to introduce automated diagnostic techniques.MethodsIn this study, we propose a computer-aided diagnostic algorithm (Hybrid-FHR) for fetal acidosis to assist physicians in making objective decisions and taking timely interventions. Hybrid-FHR uses multi-modal features, including one-dimensional FHR signals and three types of expert features designed based on prior knowledge (morphological time domain, frequency domain, and nonlinear). To extract the spatiotemporal feature representation of one-dimensional FHR signals, we designed a multi-scale squeeze and excitation temporal convolutional network (SE-TCN) backbone model based on dilated causal convolution, which can effectively capture the long-term dependence of FHR signals by expanding the receptive field of each layer’s convolution kernel while maintaining a relatively small parameter size. In addition, we proposed a cross-modal feature fusion (CMFF) method that uses multi-head attention mechanisms to explore the relationships between different modalities, obtaining more informative feature representations and improving diagnostic accuracy.ResultsOur ablation experiments show that the Hybrid-FHR outperforms traditional previous methods, with average accuracy, specificity, sensitivity, precision, and F1 score of 96.8, 97.5, 96, 97.5, and 96.7%, respectively.ConclusionsOur algorithm enables automated CTG analysis, assisting healthcare professionals in the early identification of fetal acidosis and the prompt implementation of interventions.

CSST-Net: Channel Split Spatiotemporal Network for Human Action Recognition

Temporal reasoning is crucial for action recognition tasks. The previous works use 3D CNNs to jointly capture spatiotemporal information, but it causes a lot of computational costs as well. To improve the above problems, we propose a general channel split spatiotemporal network (CSST-Net) to achieve effective spatiotemporal feature representation learning. The CSST module consists of the grouped spatiotemporal modeling (GSTM) module and the parameter-free feature fusion (PFFF) module. The GSTM module decomposes features into spatial and temporal parts along the channel dimension in parallel, which focuses on spatial and temporal clues respectively. Meanwhile, we utilize the combination of group-wise convolution and point-wise convolution to reduce the number of parameters in the temporal branch, thus alleviating the overfitting of 3D CNNs. Furthermore, for the problem of spatiotemporal feature fusion, the PFFF module performs the recalibration and fusion of spatial and temporal features by a soft attention mechanism, without introducing extra parameters, thus ensuring the correct network information flow effectively. Finally, extensive experiments on three benchmark databases (Sth-Sth V1, Sth-Sth V2, and Jester) indicate that the proposed CSST-Net can achieve competitive performance compared to existing methods, and significantly reduces the number of parameters and FLOPs of 3D CNNs baseline.

Interpretable spatio-temporal modeling for soil temperature prediction

Soil temperature (ST) is a crucial parameter in Earth system science. Accurate ST predictions provide invaluable insights; however, the “black box” nature of many deep learning approaches limits their interpretability. In this study, we present the Encoder-Decoder Model with Interpretable Spatio-Temporal Component (ISDNM) to enhance both ST prediction accuracy and its spatio-temporal interpretability. The ISDNM combines a CNN-encoder-decoder and an LSTM-encoder-decoder to improve spatio-temporal feature representation. It further uses linear regression and Uniform Manifold Approximation and Projection (UMAP) techniques for clearer spatio-temporal visualization of ST. The results show that the ISDNM model had the highest R2 ranging from 0.886 to 0.963 and the lowest RMSE ranging from 6.086 m3/m3 to 12.533 m3/m3 for different climate regions, and demonstrated superior performance than all the other DL models like CNN, LSTM, ConvLSTM models. The predictable component highlighted the remarkable similarity between Medium fine and Very fine soils in China. Additional, May and November emerged as crucial months, acting as inflection points in the annual ST cycle, shaping ISDNM model’s prediction capabilities.

Spatio-temporal multi-graph convolutional network based on wavelet analysis for vehicle speed prediction

The research on theoretical techniques and practical applications of vehicle speed prediction holds significant importance in mitigating the increasingly pressing traffic issues and enhancing the operational efficiency of urban transportation systems. This paper aims to improve the accuracy of traffic speed prediction. To achieve this, we have devised a multi-graph convolutional neural network (GCN) framework that utilizes data from three different patterns: recent, daily, and weekly, enabling the extraction of distinct time-related features. Through the integration of multiple GCNs and 3D convolutional neural networks (3DCNN), we have obtained richer spatiotemporal feature representations. Additionally, we have applied wavelet analysis to process the traffic speed data from each node in the road network, unearthing hidden information and eliminating noise interference. Our experiments have demonstrated that the model employing wavelet analysis outperforms the baseline model without wavelet analysis, thus enhancing the prediction accuracy. In conclusion, we propose a wavelet-based spatiotemporal multi-graph convolutional neural network for traffic speed prediction. Our experimental results validate the superiority of this model across different time intervals and real-world datasets, showcasing its advantages in terms of prediction accuracy and robustness. As a result, our approach offers an effective and viable solution for traffic speed prediction tasks.

Gearbox Fault Diagnosis Based on Multi-Sensor Deep Spatiotemporal Feature Representation

The vibration signal acquired by a single sensor contains limited information and is easily interfered by noise signals, resulting in the inability to fully express the operating characteristics and state of a gearbox. To address this problem, our study proposes a gearbox fault diagnosis method based on multi-sensor deep spatiotemporal feature representation. This method utilizes two vibration sensors to obtain the vibration information of the gearbox. A fault diagnosis model (PCNN–GRU) combined with a parallel convolutional neural network (PCNN) and gated recurrent unit (GRU) was used to fuse the gearbox vibration information. The parallel convolutional neural network was used to extract the spatial information of the vibration signals collected by different position sensors, and the timing information was mined through the gated recurrent unit. The deep spatiotemporal features that fuse the multi-sensor spatial and temporal information were composed. The collected multi-sensor vibration signals were directly input into the PCNN–GRU model, and an end-to-end intelligent diagnosis of the gearbox faults was realized. Finally, through experimental verification, the accuracy rate of this model can reach up to 99.92%. Compared with other models, this model has a higher diagnostic accuracy and stability.

Video Action Recognition Using Motion and Multi-View Excitation with Temporal Aggregation.

Spatiotemporal and motion feature representations are the key to video action recognition. Typical previous approaches are to utilize 3D CNNs to cope with both spatial and temporal features, but they suffer from huge computations. Other approaches are to utilize (1+2)D CNNs to learn spatial and temporal features in an efficient way, but they neglect the importance of motion representations. To overcome problems with previous approaches, we propose a novel block which makes it possible to alleviate the aforementioned problems, since our block can capture spatial and temporal features more faithfully and efficiently learn motion features. This proposed block includes Motion Excitation (ME), Multi-view Excitation (MvE), and Densely Connected Temporal Aggregation (DCTA). The purpose of ME is to encode feature-level frame differences; MvE is designed to enrich spatiotemporal features with multiple view representations adaptively; and DCTA is to model long-range temporal dependencies. We inject the proposed building block, which we refer to as the META block (or simply "META"), into 2D ResNet-50. Through extensive experiments, we demonstrate that our proposed method architecture outperforms previous CNN-based methods in terms of "Val Top-1 %" measure with Something-Something v1 and Jester datasets, while the META yielded competitive results with the Moment-in-Time Mini dataset.

Dengue Risk Forecast with Mosquito Vector: A Multicomponent Fusion Approach Based on Spatiotemporal Analysis

Dengue as an acute infectious disease threatens global public health and has sparked broad research interest. However, existing studies generally ignore the spatial dependencies involved in dengue forecast, and consideration of temporal periodicity is absent. In this work, we propose a spatiotemporal component fusion model (STCFM) to solve the dengue risk forecast issue. Considering that mosquitoes are an important vector of dengue transmission, we introduce feature factors involving mosquito abundance and spatiotemporal lags to model temporal trends and spatial distributions separately on the basis of statistical properties. Specifically, we conduct multiscale modeling of temporal dependencies to enhance the forecast capability of relevant periods by capturing the historical variation patterns of the data across different segments in the temporal dimension. In the spatial dimension, we quantify the multivariate spatial correlation analysis as additional features to strengthen the spatial feature representation and adopt the ConvLSTM model to learn spatial dependencies adequately. The final forecast results are obtained by stacking strategy fusion in ensemble learning. We conduct experiments on real dengue datasets. The results indicate that STCFM improves prediction accuracy through effective spatiotemporal feature representations and outperforms candidate models with a reasonable component construction strategy.

SC-LPR: Spatiotemporal context based LiDAR place recognition

• A spatiotemporal and semantic information fusion based LiDAR place recognition method improves recognition accuracy. • A novel feature encoding module is proposed to effectively aggregate spatiotemporal features. • Spatiotemporal information is fused without geometrical transformations between LiDAR scans. • A superior feature similarity evaluation network is proposed to predict similarity score. 3D LiDAR-based place recognition remains challenging due to ambiguities and dynamics of scenes. Place representation can be learned effectively by deep learning, but perceptual confusion is inevitable based on single-scan. To alleviate the problem, we propose a new place recognition method named SC-LPR. In this method, spatiotemporal contextual information from LiDAR scans is used to increase the capacity of features’ representation. Firstly, a semantic graph is constructed to represent the topological geometric map of each LiDAR scan. Then, an end-to-end network is designed to predict similarity, in which GRU-EdgeConv++ is proposed to learn discriminative spatiotemporal feature representation and a novel C&TNT to predict a score. We evaluate our approach on the KITTI odometry benchmark. The experimental results show that our method can effectively fuse spatiotemporal information for place recognition. Compared with the state-of-the-art methods, our approach is superior on the KITTI dataset and can achieve competitive performance. To benefit the community by serving as a benchmark for place recognition, the code of our method will be made open-source on Github 1 1 https://github.com/Daideyun/SC-LPR .

Semi-supervised generative and discriminative adversarial learning for motor imagery-based brain\u2013computer interface

Convolutional neural networks (CNNs), which can recognize structural/configuration patterns in data with different architectures, have been studied for feature extraction. However, challenges remain regarding leveraging advanced deep learning methods in BCIs. We focus on problems of small-sized training samples and interpretability of the learned parameters and leverages a semi-supervised generative and discriminative learning framework that effectively utilizes synthesized samples with real samples to discover class-discriminative features. Our framework learns the distributional characteristics of EEG signals in an embedding space using a generative model. By using artificially generated and real EEG signals, our framework finds class-discriminative spatio-temporal feature representations that help to correctly discriminate input EEG signals. It is noteworthy that the framework facilitates the exploitation of real, unlabeled samples to better uncover the underlying patterns inherent in a user’s EEG signals. To validate our framework, we conducted experiments comparing our method with conventional linear models by utilizing variants of three existing CNN architectures as generator networks and measuring the performance on three public datasets. Our framework exhibited statistically significant improvements over the competing methods. We investigated the learned network via activation pattern maps and visualized generated artificial samples to empirically justify the stability and neurophysiological plausibility of our model.

Joint attentive adaptive feature fusion network for action recognition

Joint spatio-temporal feature learning is the key to video-based action recognition. Off-the-shelf techniques mostly apply two-stream networks, and they either simply fuse the classification scores or only integrate the high-level features. However, these methods cannot learn inter-modality relationship well. We propose a joint attentive (JA) adaptive feature fusion (AFF) network, a three-stream network that improves inter-modality fusion by exploring complementary and interactive information of two modalities, RGB and optical flow. Specifically, we design an AFF block to implement layer-wise fusion between both modality network channels and feature levels, where spatio-temporal feature representations with different modalities and various levels can be fused effectively. To capture three-dimensional interaction of spatio-temporal features, we devise a JA module by incorporating the inter-dependencies learned with the spatial-channel attention mechanism and combine multi-scale attention to refine the fine-grained features. Extensive experiments on three public action recognition benchmark datasets demonstrate that our method achieves competitive results.

From Whole Video to Frames: Weakly-Supervised Domain Adaptive Continuous-Time QoE Evaluation.

Due to the rapid increase in video traffic and relatively limited delivery infrastructure, end users often experience dynamically varying quality over time when viewing streaming videos. The user quality-of-experience (QoE) must be continuously monitored to deliver an optimized service. However, modern approaches for continuous-time video QoE estimation require densely annotating the continuous-time QoE labels, which is labor-intensive and time-consuming. To cope with such limitations, we propose a novel weakly-supervised domain adaptation approach for continuous-time QoE evaluation, by making use of a small amount of continuously labeled data in the source domain and abundant weakly-labeled data (only containing the retrospective QoE labels) in the target domain. Specifically, given a pair of videos from source and target domains, effective spatiotemporal segment-level feature representation is first learned by a combination of 2D and 3D convolutional networks. Then, a multi-task prediction framework is developed to simultaneously achieve continuous-time and retrospective QoE predictions, where a quality attentive adaptation approach is investigated to effectively alleviate the domain discrepancy without hampering the prediction performance. This approach is enabled by explicitly attending to the video-level discrimination and segment-level transferability in terms of the domain discrepancy. Experiments on benchmark databases demonstrate that the proposed method significantly improves the prediction performance under the cross-domain setting.

Energy-Based Adversarial Transfer Network for Cross-Domain Fault Diagnosis of Electro-Mechanical Systems

Recently, fault diagnosis methods based on deep learning for electro-mechanical systems have achieved excellent classification performance when the training and test data satisfied independent and identical distributions. However, distribution inconsistencies frequently occur due to operating condition variations. As a promising tool for cross-domain fault diagnosis, transfer learning can exploit source domain knowledge to facilitate target domain learning. To implement cross-domain fault diagnosis for electro-mechanical systems, an energy-based adversarial transfer network is proposed in this paper to reduce the distribution discrepancies among different domain features by incorporating an energy-based discrimination strategy into the adversarial transfer learning mechanism, which contains a state recognition module, a discrepancy metric module, and a domain discrimination module. A feature extractor is designed in the state recognition module to provide effective spatio-temporal feature representations for different domains, and an unsupervised loss is constructed according to the entropy minimization principle to improve the recognition capability of the classifier for data samples in the target domain. In the discrepancy metric module, feature distribution alignment is achieved by minimizing the distribution differences between the source domain features and the target domain ones. Furthermore, an energy discriminator is developed to effectively drive adversarial learning by introducing a flexible energy function instead of an explicit probability calculation for distinguishing the feature distributions of different domains. Two bearing datasets and one circuit dataset derived from different test platforms with various loads, speeds, stresses and fault degrees are investigated to verify the merits of the proposed model under variable operating conditions. The experimental results indicate that the proposal is superior to other state-of-the-art methods.

Improving human action recognition by jointly exploiting video and WiFi clues

Recent years have witnessed the increasing attentions on human action recognition(HAR). Traditional methods are prone to explore the optimum spatiotemporal feature representation of human actions in video clips so as to achieve high recognition performance. However, the optical limitations, such as inappropriate view, dim illumination and object occlusion, usually degrade video quality and affect the recognition performance a lot. Considering that wireless signals are robust against optical limitations, we thus incorporate WiFi signals with video streams for HAR. Specifically, we use WiFi Channel State Information as a compensator for video streams. A great challenge is how to effectively fuse the video and WiFi information to achieve better prediction performance. To this end, we employ convolution neural networks and statistic analysis algorithms to extract video and WiFi features respectively, and propose a novel multi-modal learning approach for video and WiFi feature fusion, where the video and WiFi features are projected to a common space by supervised learning. The experimental results indicate that the recognition precision of human actions in videos improved obviously with the aid of WiFi signals and the proposed multi-modal learning approach rivals the state of art methods.

STRNet: Triple-stream Spatiotemporal Relation Network for Action Recognition

Learning comprehensive spatiotemporal features is crucial for human action recognition. Existing methods tend to model the spatiotemporal feature blocks in an integrate-separate-integrate form, such as appearance-and-relation network (ARTNet) and spatiotemporal and motion network (STM). However, with blocks stacking up, the rear part of the network has poor interpretability. To avoid this problem, we propose a novel architecture called spatial temporal relation network (STRNet), which can learn explicit information of appearance, motion and especially the temporal relation information. Specifically, our STRNet is constructed by three branches, which separates the features into 1) appearance pathway, to obtain spatial semantics, 2) motion pathway, to reinforce the spatiotemporal feature representation, and 3) relation pathway, to focus on capturing temporal relation details of successive frames and to explore long-term representation dependency. In addition, our STRNet does not just simply merge the multi-branch information, but we apply a flexible and effective strategy to fuse the complementary information from multiple pathways. We evaluate our network on four major action recognition benchmarks: Kinetics-400, UCF-101, HMDB-51, and Something-Something v1, demonstrating that the performance of our STRNet achieves the state-of-the-art result on the UCF-101 and HMDB-51 datasets, as well as a comparable accuracy with the state-of-the-art method on Something-Something v1 and Kinetics-400.

Human action recognition using three orthogonal planes with unsupervised deep convolutional neural network

Deep learning models have attained great success for an extensive range of computer vision applications including image and video classification. However, the complex architecture of the most recently developed networks imposes certain memory and computational resource limitations, especially for human action recognition applications. Unsupervised deep convolutional neural networks such as PCANet can alleviate these limitations and hence significantly reduce the computational complexity of the whole recognition system. In this work, instead of using 3D convolutional neural network architecture to learn temporal features of video actions, the unsupervised convolutional PCANet model is extended into (PCANet-TOP) which effectively learn spatiotemporal features from Three Orthogonal Planes (TOP). For each video sequence, spatial frames (XY) and temporal planes (XT and YT) are utilized to train three different PCANet models. Then, the learned features are fused after reducing their dimensionality using whitening PCA to obtain spatiotemporal feature representation of the action video. Finally, Support Vector Machine (SVM) classifier is applied for action classification process. The proposed method is evaluated on four benchmarks and well-known datasets, namely, Weizmann, KTH, UCF Sports, and YouTube action datasets. The recognition results show that the proposed PCANet-TOP provides discriminative and complementary features using three orthogonal planes and able to achieve promising and comparable results with state-of-the-art methods.

DTR-HAR: deep temporal residual representation for human activity recognition

Human activity recognition (HAR) is a highly prized application in the pattern recognition and the computer vision fields. Up till now, deep neural networks have acquired big attention in computer studies and image processing fields, and have generated significant results. In this paper, we propose a deep temporal residual system for daily living activity recognition that aims to enhance spatiotemporal feature representation in order to improve the HAR system performance. To this end, we adopt a deep residual convolutional neural network (RCN) to retain discriminative visual features relayed to appearance and long short-term memory neural network to capture the long-term temporal evolution of actions. The latter was considered to implement time dependencies occurring when carrying out the activity to enhance features extracted from the RCN network by adding time information to address the dynamic activity recognition problem as a sequence labeling job. The deep temporal residual model for human activity recognition system is performed on two benchmark publicly available datasets: MSRDailyActivity3D and CAD-60. the proposed system achieves very competitive results when compared to others from the state of the art.