Spatial-temporal Features Research Articles

STFEformer: Spatial–Temporal Fusion Embedding Transformer for Traffic Flow Prediction

In the realm of Intelligent Transportation Systems (ITSs), traffic flow prediction is crucial for multiple applications. The primary challenge in traffic flow prediction lies in the handling and modeling of the intricate spatial–temporal correlations inherent in transport data. In recent years, many studies have focused on developing various Spatial–Temporal Graph Neural Networks (STGNNs), and researchers have also begun to explore the application of transformers to capture spatial–temporal correlations in traffic data. However, GNN-based methods mainly focus on modeling spatial correlations statically, which significantly limits their capacity to discover dynamic and long-range spatial patterns. Transformer-based methods have not sufficiently extracted the comprehensive representation of traffic data features. To explore dynamic spatial dependencies and comprehensively characterize traffic data, the Spatial–Temporal Fusion Embedding Transformer (STFEformer) is proposed for traffic flow prediction. Specifically, we propose a fusion embedding layer to capture and fuse both native information and spatial–temporal features, aiming to achieve a comprehensive representation of traffic data characteristics. Then, we introduce a spatial self-attention module designed to enhance detection of dynamic and long-range spatial correlations by focusing on interactions between similar nodes. Extensive experiments conducted on three real-world datasets demonstrate that STFEformer significantly outperforms various baseline models, notably achieving up to a 5.6% reduction in Mean Absolute Error (MAE) on the PeMS08 dataset compared to the next-best model. Furthermore, the results of ablation experiments and visualizations are employed to clarify and highlight our model’s performance. STFEformer represents a meaningful advancement in traffic flow prediction, potentially influencing future research and applications in ITSs by providing a more robust framework for managing and analyzing traffic data.

TSANN-TG: Temporal-Spatial Attention Neural Networks with Task-Specific Graph for EEG Emotion Recognition.

Electroencephalography (EEG)-based emotion recognition is increasingly pivotal in the realm of affective brain-computer interfaces. In this paper, we propose TSANN-TG (temporal-spatial attention neural network with a task-specific graph), a novel neural network architecture tailored for enhancing feature extraction and effectively integrating temporal-spatial features. TSANN-TG comprises three primary components: a node-feature-encoding-and-adjacency-matrices-construction block, a graph-aggregation block, and a graph-feature-fusion-and-classification block. Leveraging the distinct temporal scales of features from EEG signals, TSANN-TG incorporates attention mechanisms for efficient feature extraction. By constructing task-specific adjacency matrices, the graph convolutional network with an attention mechanism captures the dynamic changes in dependency information between EEG channels. Additionally, TSANN-TG emphasizes feature integration at multiple levels, leading to improved performance in emotion-recognition tasks. Our proposed TSANN-TG is applied to both our FTEHD dataset and the publicly available DEAP dataset. Comparative experiments and ablation studies highlight the excellent recognition results achieved. Compared to the baseline algorithms, TSANN-TG demonstrates significant enhancements in accuracy and F1 score on the two benchmark datasets for four types of cognitive tasks. These results underscore the significant potential of the TSANN-TG method to advance EEG-based emotion recognition.

Video saliency prediction for First-Person View UAV videos: Dataset and benchmark

Visual saliency prediction plays a crucial role in Unmanned Aerial Vehicle (UAV) video analysis tasks. In this paper, an eye-tracking dataset of the immersive viewing of videos captured from a First-Person View (FPV) of UAVs is developed, which consists of 200 video clips captured by DJI FPV drones, with a resolution of 4K QHD. The videos cover six different genres and fourteen unique scenes. To study human visual attention in watching FPV videos, fixation points are recorded using an eye tracker integrated into a VR headset. Based on the dataset, a simple yet effective FPV UAV video Saliency prediction model (FUAVSal) is proposed as a baseline, considering spatial–temporal feature, camera motion information and FPV prior. To establish benchmarks for saliency prediction in immersive FPV UAV video viewing, sixteen computational models are evaluated on this dataset. Detailed quantitative and qualitative comparisons are provided. The developed dataset and benchmarks aim to facilitate research on visual saliency prediction for First-Person View UAV videos.

Predicting and compensating for small-sample thermal information data in precision machine tools: A spatial-temporal interactive integration network and digital twin system approach

The thermal errors in ball screws present complex spatial-temporal characteristics and exhibit persistent long-term memory, significantly affecting the machining accuracy of the whole machine tool in applications. Deep learning has emerged as a promising approach to predict these errors using thermal data. However, the scarcity of extensive thermal datasets, typically due to the prohibitive costs and time required for experimental acquisition, hampers the comprehensive exploitation of spatial-temporal and long-term memory attributes. Moreover, the current thermal error compensation system does not support the deployment of deep learning-based thermal error models due to its weak real-time performance. These limitations lead to suboptimal prediction accuracy and inadequate error compensation. Addressing this challenge, a novel spatial-temporal interactive integration network is proposed, and this is a sophisticated model that synergistically blends a time memory gate and a spatial-temporal fusion gate within an advanced attention-based spatial-temporal graph convolutional framework. Our network ingeniously leverages spatial data to anchor long-term memory while employing temporal data for selective memory feature extraction, facilitating a refined integration of spatial-temporal features. Concurrently, the integration of gated recurrent units and a time attention layer meticulously extracts and enhances temporal features, bolstered by our innovative time memory gate, which is adept at handling small-sample scenarios and refining thermal error predictions. Critically, the digital twin system for thermal error compensation is constructed to improve the system’s real-time performance. The integration of this network into the digital twin system marks a pivotal advancement in thermal error compensation. Our empirical results underscore the system's remarkable robustness and superior predictive accuracy, demonstrating a significant reduction in positioning and machining errors (over 90% and 80% respectively) even with constrained thermal data inputs. These advancements not only delineate a substantial leap in predictive accuracy but also underscore our contribution to reducing operational costs and enhancing the efficacy of machine tools.

Adaptive Temporal-Spatial Pyramid Variational Autoencoder Model for Multirate Dynamic Chemical Process Soft Sensing Application.

Data-driven soft sensors play an important role in practical processes and have been widely applied. They provide real-time prediction of quality variables and then guide production and improve product quality. In practical chemical production processes, nonlinear dynamic multirate data is widespread and challenging to model. This paper innovatively proposes a temporal-spatial pyramid variational autoencoder (TS-PVAE) model for the nonlinear temporal-spatial feature pyramid extraction from multirate data. This structure not only selectively utilizes multirate data but also handles complex nonlinear time-series data. Based on this, integrated with just-in-time (JIT) learning, an adaptive TS-PVAE (ATS-PVAE) model is developed. In this model, historical data are used for real-time fine-tuning of the model, leading to the development of an adaptive model. Finally, the proposed models are validated by an industrial case of a methanation furnace, demonstrating a superior estimation performance.

Automatic multi-label diagnosis of single-lead ECG using novel hybrid residual recurrent convolutional neural networks

Arrhythmia is a prevalent cardiovascular disease that can unveil various heart health issues. In recent times, the rise of wearable devices has garnered attention towards the advantages of portable single-lead ECG devices in heart health monitoring. Achieving end-to-end ECG automatic diagnosis through deep learning is essential due to the time-consuming and labour-intensive characteristics of manual diagnosis of ECG records. This paper presents the design of a hybrid residual recurrent convolutional neural network (HRRCNN) for automatically diagnosing single-lead ECG with multiple labels. Firstly, the convolutional layers and residual networks extract the spatial and short-term temporal features. Subsequently, a hybrid recurrent neural network (RNN) comprised of a double-layer bidirectional gated recurrent unit (GRU) and a bidirectional long short-term memory (LSTM) is employed to capture long and short-term temporal features. Finally, an output layer, comprising an attention layer and a fully connected layer, converts the extracted features into classification results. To evaluate the performance of HRRCNN, a dataset from the Shanghai First People’s Hospital Affiliated to Shanghai JiaoTong University(Shanghai General Hospital, SGH) is utilized. The experimental results demonstrate that HRRCNN achieves an accuracy of 95.08 %, F1-macro of 87.22 %, Sensitivity of 88.51 %, and a positive prediction rate of 86.13 %. Comparative analysis with existing methods underscores the superior performance of HRRCNN, thereby establishing its potential as an out-of-hospital heart health monitoring tool and an auxiliary tool in clinical settings.

Spatial–Temporal Analysis-Based Video Quality Assessment: A Two-Stream Convolutional Network Approach

In system processing, video inevitably suffers from distortion, which leads to quality degradation and affects the user experience. Therefore, it is of great importance to design an accurate and effective objective video quality assessment (VQA) method. In this paper, by considering the multi-dimensional characteristics for video and visual perceptual mechanism, a two-stream convolutional network for VQA is proposed based on spatial–temporal analysis, named TSCNN-VQA. Specifically, for feature extraction, TSCNN-VQA first extracts spatial and temporal features by two different convolutional neural network branches, respectively. After that, the spatial–temporal joint feature fusion is constructed to obtain the joint spatial–temporal features. Meanwhile, the TSCNN-VQA also integrates an attention module to guarantee that the process conforms to the mechanism that the visual system perceives video information. Finally, the overall quality is obtained by non-linear regression. The experimental results in both the LIVE and CSIQ VQA datasets show that the performance indicators obtained by TSCNN-VQA are higher than those of existing VQA methods, which demonstrates that TSCNN-VQA can accurately evaluate video quality and has better consistency with the human visual system.

Incorporating the Third Law of Geography with Spatial Attention Module–Convolutional Neural Network–Transformer for Fine-Grained Non-Stationary Air Quality Predictive Learning

Accurate air quality prediction is paramount in safeguarding public health and addressing air pollution control. However, previous studies often ignore the geographic similarity among different monitoring stations and face challenges in dynamically capturing different spatial–temporal relationships between stations. To address this, an air quality predictive learning approach incorporating the Third Law of Geography with SAM–CNN–Transformer is proposed. Firstly, the Third Law of Geography is incorporated to fully consider the geographical similarity among stations via a variogram and spatial clustering. Subsequently, a spatial–temporal attention convolutional network that combines the spatial attention module (SAM) with the convolutional neural network (CNN) and Transformer is designed. The SAM is employed to extract spatial–temporal features from the input data. The CNN is utilized to capture local information and relationships among each input feature. The Transformer is applied to capture time dependencies across long-distance time series. Finally, Shapley’s analysis is employed to interpret the model factors. Numerous experiments with two typical air pollutants (PM2.5, PM10) in Haikou City show that the proposed approach has better comprehensive performance than baseline models. The proposed approach offers an effective and practical methodology for fine-grained non-stationary air quality predictive learning.

Enhanced Multi-Task Traffic Forecasting in Beyond 5G Networks: Leveraging Transformer Technology and Multi-Source Data Fusion

Managing cellular networks in the Beyond 5G (B5G) era is a complex and challenging task requiring advanced deep learning approaches. Traditional models focusing on internet traffic (INT) analysis often fail to capture the rich temporal and spatial contexts essential for accurate INT predictions. Furthermore, these models do not account for the influence of external factors such as weather, news, and social trends. This study proposes a multi-source CNN-RNN (MSCR) model that leverages a rich dataset, including periodic, weather, news, and social data to address these limitations. This model enables the capture and fusion of diverse data sources for improved INT prediction accuracy. An advanced deep learning model, the transformer-enhanced CNN-RNN (TE-CNN-RNN), has been introduced. This model is specifically designed to predict INT data only. This model demonstrates the effectiveness of transformers in extracting detailed temporal-spatial features, outperforming conventional CNN-RNN models. The experimental results demonstrate that the proposed MSCR and TE-CNN-RNN models outperform existing state-of-the-art models for traffic forecasting. These findings underscore the transformative power of transformers for capturing intricate temporal-spatial features and the importance of multi-source data and deep learning techniques for optimizing cell site management in the B5G era.

Decoupled and boosted learning for skeleton-based dynamic hand gesture recognition

With the development of cost-effective depth sensors, skeleton-based dynamic hand gesture recognition has made significant progress. Existing methods mostly utilize a single model to learn all spatial–temporal features. Meanwhile, they cannot effectively boost key features and make use of multi-scale features. In this paper, we propose a lightweight dual-stream framework, which consists of a temporal mutual boosted stream (TMB-Stream) and a spatial self-boosted stream (SSB-Stream). In the TMB-Stream, we design a hybrid attention module (HAM) to boost important motion features from temporal sequences, which is composed of a multi-scale multi-head attention module (MMAM) and a spatial–temporal attention module (STAM). In the SSB-Stream, we present a self-boosted learning manner to promote the performance of the spatial stream. Specifically, we design a multi-scale auto-encoder (MAE), which can use limited skeleton data to extract and boost spatial latent features by minimizing the gap between original and reconstructed skeleton images. In addition, we propose a multi-scale fusion module (MFM) to effectively fuse multi-scale features in stages. Experimental results show that our lightweight framework yields satisfactory performance on SHREC’17 Track and DHG-14/28 datasets, as well as very competitive performance on FPHA dataset.

Spatial continuity and stress transfer: Primary and complementary factors shaping pure-swarm laboratory catalog into mixed burst-like and swarm-like

Natural earthquakes and micro-seismicity resulting from hydraulic fracturing or other engineering practices display distinctively different spatial-temporal features, like mixed burst- and swarm-like features or predominantly swarm-like features. The mechanism(s) contributing to such observations can be diverse. We present the inspections on the dynamic formation process of the single swarm-like tree in laboratory acoustic emission (AE) catalogs. Such largest swarm-like trees can contain >97% AE events from the entire catalog within a test; and all catalogs under investigation display scale-invariance features. The formation of the largest swarm-like tree correlates with the rock fracture process analogue of the source pervasive process, where its AE releases exhibit significant spatial well-organization. Comparison to other laboratory catalogs under different laboratory settings helps us identify the spatial continuity of the rock fracture process as the primary factor in forming the largest swarm-like trees at laboratory scale. The stress transfer process is involved in the rock fracture process for the tests having pre-existing spatial discontinuity. Artificial perturbations on the spatial information induced by the stress transfer process further confirm that stress transfer also serves to shift the pure swarm-like catalog into a mixed burst- and swarm-like catalog. These laboratory observations may provide inspirational insights for understanding the field-scale mechanism(s) shaping the spatial-temporal energy release features.

Quantitative Assessment via Multi-Domain Fusion of Muscle Synergy Associated With Upper-Limb Motor Function for Stroke Rehabilitation.

Quantitative assessment of upper limb motor function aids therapists in providing appropriate rehabilitation strategies, which plays an essential role in post-stroke rehabilitation. Traditional assessments, relying on clinical scales or kinematic metrics, often involve subjective scores or are influenced by compensatory strategies. Recently, the use of muscle synergies, representing simplified neuromuscular control, has emerged as a promising approach for post-stroke assessment. In general, muscle synergies are decomposed into two components: synergy vectors and synergy activation. Synergy vectors represent the relative weighting of each muscle within each synergy, that is muscle coordination; synergy activation represents the recruitment of the muscle synergy over time, that is muscle activation strength. Both components are vital for adequately assessing patients' motor function. Therefore, we integrate the spatial domain and temporal domain features extracted from synergy vectors and synergy activation, constructing a multi-domain assessment system using a Random Forest classifier, which may provide great qualitative classification accuracy. Furthermore, a novel functional score is generated from the probabilities belonging to the pathological group. Finally, A study involving ten healthy subjects and ten post-stroke patients validates the proposed method. The experimental results show that the classification accuracy was enhanced to 98.56% by fusing the characteristics derived from different domains, which was higher than that based on spatial domain (94.90%) and temporal domain (91.08%), respectively. Furthermore, the assessment score generated by multi-domain fusion framework exhibited a significant correlation with the clinical score. These promising results show the potential of applying the proposed method to clinical assessments for post-stroke patients.

A brain topography graph embedded convolutional neural network for EEG-based motor imagery classification

Brain-computer interface (BCI) based on motor imagery EEG (MI-EEG) has been used extensively in health care, device control, entertainment, and other fields. MI-EEG signals consist of multiple channels that exhibit a specific topological relationship. However, many existing methods based on deep learning do not make good use of the multi-electrode characteristics of EEG signals. To further improve the decoding performance of MI-EEG, this study proposes a novel method combining the brain topology graph embedding and convolutional neural network (CNN) for MI-EEG classification. Specifically, mutual information is used to measure the relationship between EEG channels. A graph convolutional network (GCN) is employed to embed the topological relationship. Following this, a convolutional block using depth-wise convolution is built to extract spatial–temporal features, while a temporal convolutional network (TCN) is used to extract advanced temporal features. The model’s performance is evaluated on BCICIV-2a and HGD datasets. The experimental results show that the classification accuracy of the proposed model is 80.46% and 94.38%, respectively, outperforming existing methods. Moreover, ablation experiments and feature visualization show that graph embedding can significantly improve the decoding performance of MI-EEG.

Recognition of EEG-based movement intention combined with channel selection adopting deep learning methods

Brain-computer interface (BCI) is an emerging technology which provides a road to control communication and external devices. Electroencephalogram (EEG)-based motor imagery (MI) tasks recognition has important research significance for stroke, disability and others in BCI fields. However, enhancing the classification performance for decoding MI-related EEG signals presents a significant challenge, primarily due to the variability across different subjects and the presence of irrelevant channels. To address this issue, a novel hybrid structure is developed in this study to classify the MI tasks via deep separable convolution network (DSCNN) and bidirectional long short-term memory (BLSTM). First, the collected time-series EEG signals are initially processed into a matrix grid. Subsequently, data segments formed using a sliding window strategy are inputted into proposed DSCNN model for feature extraction (FE) across various dimensions. And, the spatial-temporal features extracted are then fed into the BLSTM network, which further refines vital time-series features to identify five distinct types of MI-related tasks. Ultimately, the evaluation results of our method demonstrate that the developed model achieves a 98.09% accuracy rate on the EEGMMIDB physiological datasets over a 4-second period for MI tasks by adopting full channels, outperforming other existing studies. Besides, the results of the five evaluation indexes of Recall, Precision, Test-auc, and F1-score also achieve 97.76%, 97.98%, 98.63% and 97.86%, respectively. Moreover, a Gradient-class Activation Mapping (GRAD-CAM) visualization technique is adopted to select the vital EEG channels and reduce the irrelevant information. As a result, we also obtained a satisfying outcome of 94.52% accuracy with 36 channels selected using the Grad-CAM approach. Our study not only provides an optimal trade-off between recognition rate and number of channels with half the number of channels reduced, but also it can also advances practical application research in the field of BCI rehabilitation medicine, effectively.

PKG-DTSFLN: Process Knowledge-guided Deep Temporal–spatial Feature Learning Network for anode effects identification

In the aluminum electrolysis process, the accurate identification of anode effect (AE) can improve production efficiency. However, the existing methods fail to effectively capture the features of the anode current signal (ACS) due to its complex dynamic characteristics and temporal–spatial dependence. To address this issue, we propose a Process Knowledge-guided Deep Temporal–spatial Feature Learning Network (PKG-DTSFLN). We believe that knowledge and production data are complementary. Knowledge has potential to deduce beyond observational conditions. Data can be used to detect unexpected patterns. The combination of data and knowledge is potential to improve the performance. Specifically, knowledge is utilized to construct the adjacency matrix to represent the spatial structure of ACS. Then, a deep learning model is constructed by integrating the 1D-CNN and GAT, which is used to capture the temporal–spatial features of ACS. The experimental results on ACS dataset show that the accuracy is more than 99% with low computational cost.

Mapping tobacco planting areas in smallholder farmlands using Phenological-Spatial-Temporal LSTM from time-series Sentinel-1 SAR images

Phenological information on crop growth aids in identifying crop types from remote sensing images, but its incorporation into classification models is insufficiently exploited, especially in deep learning frameworks. This study presents a new model, Phenological-Temporal-Spatial LSTM (PST-LSTM), for mapping tobacco planting areas in smallholder farmlands using time-series Sentinel-1 Synthetic Aperture Radar (SAR) images. The PST-LSTM model is built on a multi-modal learning framework that fuses phenological information with deep spatial–temporal features. We applied the model to extract tobacco planting areas in Ninghua, Pucheng, and Shanghang Counties, in Fujian Province, and Luoping County in Yunnan Province, China. We compared PST-LSTM with existing methods based on phenological rules and Dynamic Time Warping (DTW) methods, and analyzed its strength in feature fusion. Results showed that our model outperformed these methods, achieving an overall accuracy (OA) of 93.16% compared to 86.69% and 85.93% for the phenological rules and DTW methods, respectively, in the Ninghua area. PST-LSTM effectively integrated time-series data with phenological information derived from different strategies at the feature level and performed better than existing feature fusion methods (based upon fuzzy sets) at the decision level. It also demonstrated a better spatial transferability than other methods when applied to different study areas, achieving an OA of 90.95%, 91.41%, and 80.75% for the Pucheng, Shanghang, and Luoping areas, respectively, using training samples from Ninghua. We conclude that PST-LSTM can effectively extract tobacco planting areas in smallholder farming from time-series SAR images and has the potential for mapping other crop types.

Parallel framework of a multi-graph convolutional network and gated recurrent unit for spatial–temporal metro passenger flow prediction

Metro passenger flow prediction is a critical problem in metro transport systems. However, recent studies have either overlooked spatial information on the metro network or primarily focused on modeling spatial dependencies using only the physical topology. To achieve accurate metro passenger flow (inflow and outflow at each station of a network) prediction, this study proposes a joint prediction model that combines the multi-graph convolution network and the gated recurrent unit (GRU). In addition to exploring location topology relationships, the proposed model selects two non-Euclidean spatial dependencies in metro passenger flow prediction to design essential graph elements as part of the stacked spatial block. Three spatial relationships (adjacency, similarity, and correlation) are integrated in parallel with the GRU network. The metro passenger flow prediction framework ASC-GRU (adjacency, similarity, correlation, and gated recurrent unit) is designed to mitigate the distortion of results during the capturing of passenger flow spatial–temporal features. Finally, ASC-GRU is tested using two datasets from the Hangzhou and Shanghai metro networks in China, and the error metrics of different models are compared and analyzed to verify the effectiveness and feasibility of ASC-GRU. The test results demonstrate that the proposed model outperforms other baseline models in passenger flow prediction over long time intervals and large networks. In particular, compared with the best performance of the baselines, the average reduction is around 3%, 12% and 13% in metrics of MAPE, MAE and RMSE, respectively.

Research on Knowledge Tracing-Based Classroom Network Characteristic Learning Engagement and Temporal-Spatial Feature Fusion

To accurately assess students’ cognitive state of knowledge points in the learning process within the smart classroom, a knowledge tracing (KT) model based on classroom network characteristic learning engagement and temporal-spatial feature fusion (CL-TSKT) is proposed. First, a classroom network is constructed based on the information of the student ID, seating relationship, student–student interaction, head-up or head-down state, and classroom network characteristics obtained from a smart classroom video. Second, a learning engagement model is established by utilizing the student–student interactions, head-up or head-down state, and classroom network characteristics. Finally, according to the learning engagement model and the knowledge point test data, a parallel temporal attention GRU network is proposed. It is utilized to extract the temporal features of the knowledge points and learning engagement. They are fused to obtain the knowledge point-learning engagement temporal characteristics and their associated attributes. Meanwhile, a CNN is used to extract the knowledge point-knowledge point spatial features. We consider the associative properties of knowledge point-knowledge points from a spatial perspective and fuse the knowledge point-knowledge point spatial features with the knowledge point-learning engagement temporal features. To accurately characterize the cognitive state of the knowledge points and provide effective support for teachers’ accurate and sustainable interventions for learners in the teaching and learning process, this paper conducts extensive experiments on four real datasets. The CL-TSKT model in this paper shows superior performance in all four evaluation metrics, compared with the state-of-the-art KT models.

AugLPN-NILM: Augmented lightweight parallel network for NILM embedding attention module over sequence to point

Non-Intrusive Load Monitoring (NILM), also called energy disaggregation, is a feasible strategy for identifying appliance-by-appliance power consumption given the aggregated power demand of electrical loads. A non-intrusive sequence to point load disaggregation parallel model (AugLPN-NILM) is proposed in this paper. Spatial-temporal feature fusion is introduced in our AugLPN to offer more differentiated features, which alleviates the weakness of time domain feature loss, therefore, can boost the performance of energy disaggregation. Most current methods have large performance variations of predicting power consumption for different appliances, the parallel architecture in the AugLPN not only greatly reduces the variations with a relatively small number of parameters to tune, but also accurately predicts the on/off state of appliances across unseen houses or buildings. Furthermore, we present an enhanced attention module, which is applied to activate or restrain features adaptively and especially improve decomposition accuracy of type II appliances. In addition, an effective postprocessing algorithm is proposed to further optimize the disaggregation results. The experiments executed on two publicly accessible datasets, UK-DALE and REDD, demonstrate our proposed AugLPN model’s superiority compared with some existing algorithms in all the identical experimental conditions. The exact code for reproducing the results in this paper is available at https://github.com/linfengYang/AugLPN_NILM.

Deep Fusion Module for Video Action Recognition

In video action recognition, effective spatiotemporal modeling is crucial. However, traditional two-stream methods face challenges in integrating spatial information from RGB images and temporary information from optical flow without long-range temporal modelling. To address these limitations, we propose the Deep Fusion Module (DFM), which focuses on the deep fusion of spatial and temporal information and consists of two components. First, we propose an Attention Fusion Module (AFM) to effectively fuse the shallow features obtained from a two-stream network, thereby facilitating the integration of spatial and temporal information. Next, we incorporate a SpatioTemporal Module (STM), comprising a ConvGRU and a 1×1 convolution, to model long-range temporal dependency and fuse spatial-temporal features. Experiments on the UCF101 dataset show that our method achieves 96.5% accuracy, outperforming baseline two-stream models by 0.3%.