Extract Temporal Features Research Articles

Enhancing lifestyle and health monitoring of elderly populations using CSA-TkELM classifier

Nowadays, there is a growing interest among researchers in health monitoring using body sensor data due to its usage in diverse applications. Human Activity Recognition (HAR) utilizes smart wearable devices, particularly in healthcare, to help the elderly population improve their lifestyle by preventing forthcoming exceptional events such as health risks and falls. Although various types of sensors are available to recognize human behaviors, wearable sensors provide consistent information about the lifestyle and functionality of subjects. However, accurate recognition requires intelligent algorithms to extract temporal features of time series data from raw sensor readings. In order to recognize human activities more accurately from time series data, this paper proposes a crossover Aquila-optimized Tri-kernel Extreme Learning Machine (CSA-TkELM) classifier. The sensor data utilized to evaluate the effectiveness of the proposed CSA-TkELM classifier is obtained from the USC Human Activity Dataset (USC-HAD), the Public domain dataset for Real-life human activity recognition using smartphone Sensors (PDS-SS), and the OPPORTUNITY dataset. The direct application of raw time series data to the classifier affects its recognition performance. Therefore, preprocessing steps such as noise removal, handling missing values, standardization, and segmentation are executed. The proposed CSA-TkELM classifier extracts significant temporal features from the time series information and recognizes activities. Several performance metrics, such as recall, precision, accuracy, and F1 score, are measured to investigate the results of the proposed CSA-TkELM classifier. The proposed CSA-TkELM classifier achieves a recall of 98.1%, precision of 97.6%, accuracy of 97.9%, and F1 score of 97.3%, which are higher than those of other compared existing techniques.

Short‐term traffic flow prediction based on AST‐MTL‐CNN‐GRU

AbstractTo address the current objective of short‐term traffic flow prediction on roads, this paper aims to predict not only the future traffic flow on roads, but also the average speed of future roads. Specifically, the authors propose a short‐term traffic flow prediction method based on the Convolutional Neural Network—Gate Recurrent Unit—Multi Task Learning‐time‐separated attention mechanism (AST‐MTL‐CNN‐GRU). Firstly, the authors add a time‐separated attention mechanism that can effectively generate attention matrices independently for different locations and times of the day. Secondly, the three main parameters of traffic flow, average speed and occupancy are combined in pairs as input data while the multi‐task learning is applied to predict the future traffic flow and average speed on the road. Finally, the CNNs are used to extract the spatial features and the GRUs are used to extract temporal features. Compared with the baseline models (diffusion CNNs (DCRNN), 1D‐CNN‐long short‐term memory (1DCNN‐LSTM), convolutional LSTM (ConvLSTM), graph WaveNet and ST‐ResNet), the authors’ proposed model has high performance in terms of accuracy, stability and robustness. A large number of ablation experiments are conducted based on five‐fold cross validation using the Quanzhou city dataset and the California dataset. The experimental results show that the combination of average speed and occupancy for input has higher prediction accuracy. In addition, the authors’ proposed model can outperform the other models being compared in terms of accuracy, stability and robustness even with noise added.

Extracting Semantic-Dynamic Features for Long-Term Stable Brain Computer Interface

Brain-computer Interface (BCI) builds a neural signal to the motor command pathway, which is a prerequisite for the realization of neural prosthetics. However, a long-term stable BCI suffers from the neural data drift across days while retraining the BCI decoder is expensive and restricts its application scenarios. Recent solutions of neural signal recalibration treat the continuous neural signals as discrete, which is less effective in temporal feature extraction. Inspired by the observation from biologists that low-dimensional dynamics could describe high-dimensional neural signals, we model the underlying neural dynamics and propose a semantic-dynamic feature that represents the semantics and dynamics in a shared feature space facilitating the BCI recalibration. Besides, we present the joint distribution alignment instead of the common used marginal alignment strategy, dealing with the various complex changes in neural data distribution. Our recalibration approach achieves state-of-the-art performance on the real neural data of two monkeys in both classification and regression tasks. Our approach is also evaluated on a simulated dataset, which indicates its robustness in dealing with various common causes of neural signal instability.

Power grid investment forecasting method based on PSO‐GRU neural network

SummaryTraditional power grid investment forecasting models often ignore the cyclical characteristics of historical investment data, leading to one‐sided investment allocation results and insufficient model generalization ability. This article proposed a power network investment forecasting method based on particle swarm optimization‐gate recurrent unit (PSO‐GRU) neural network. First, the temporal attention mechanism is introduced into the traditional GRU network, which improves the ability of the network to extract temporal features. Then, in order to avoid the adverse effects of unreasonable parameter configuration on model training, an optimized particle swarm optimization algorithm was proposed to optimize the parameter set of GRU and improve the training accuracy of the model. The data provided by an electric power company in China is used for experimental analysis, and the predicted results are compared with other electric power investment models. The RMSE of the PSO‐GRU model proposed in this article is 0.1223, which is superior to other algorithms and has certain effectiveness.

Weighted knowledge distillation of attention-LRCN for recognizing affective states from PPG signals

The recognition of affective states is important for regulating stress levels and maintaining mental health, and it is known that affective states can be inferred from physiological signals. However, in practice, the problem of affect recognition contains many challenges due to various types of external noise and different individual characteristics. This paper proposes a deep learning model called Attention-LRCN for recognizing affective states from photoplethysmography (PPG) signals. We construct a long-term recurrent convolutional network to extract temporal features from spectrograms, and a novel attention module is introduced to alleviate the effect of noise components in PPG signals. Moreover, to improve the recognition accuracy, we propose a weighted knowledge distillation technique, which is a teacher–student learning framework. We quantify the uncertainty of teacher’s predictions, and the predictive uncertainty is utilized to adaptively compute the weight of the distillation loss. To demonstrate the effectiveness of the proposed method, experiments were conducted on the WESAD dataset, which is a public dataset for stress and affect detection. We also collected our own dataset from 34 subjects to verify the accuracy of the proposed method. Experimental results demonstrate that the proposed method significantly outperforms previous algorithms on both the public and real-world datasets. The code is available at https://github.com/ziiho08/Attention-LRCN.

Suspicious Behavior Detection with Temporal Feature Extraction and Time-Series Classification for Shoplifting Crime Prevention.

The rise in crime rates in many parts of the world, coupled with advancements in computer vision, has increased the need for automated crime detection services. To address this issue, we propose a new approach for detecting suspicious behavior as a means of preventing shoplifting. Existing methods are based on the use of convolutional neural networks that rely on extracting spatial features from pixel values. In contrast, our proposed method employs object detection based on YOLOv5 with Deep Sort to track people through a video, using the resulting bounding box coordinates as temporal features. The extracted temporal features are then modeled as a time-series classification problem. The proposed method was tested on the popular UCF Crime dataset, and benchmarked against the current state-of-the-art robust temporal feature magnitude (RTFM) method, which relies on the Inflated 3D ConvNet (I3D) preprocessing method. Our results demonstrate an impressive 8.45-fold increase in detection inference speed compared to the state-of-the-art RTFM, along with an F1 score of 92%,outperforming RTFM by 3%. Furthermore, our method achieved these results without requiring expensive data augmentation or image feature extraction.

A multi-task hybrid emotion recognition network based on EEG signals

Electroencephalogram (EEG)-based emotion recognition is getting a lot of attention in Brain–Computer Interface (BCI) research. However, the recognition model of cross-subject emotions requires effectively represented EEG data to capture diverse features corresponding to the spatial, frequency and temporal of information. In contrast, by calculating Power Spectral Density (PSD), Differential Entropy (DE), and the spaced interpolation algorithm, brain map construction has the ability to enhance the quality of frequency and spatial data features. In addition, DEAP is a dimensional dataset. We used the multi-task learning method while considering the characteristics of the dimensional dataset. Therefore, we proposed a novel hybrid emotion recognition deep learning network called the multi-task hybrid emotion recognition network(MTHNet), which fuses Bi-LSTM(Bidirectional long short-term memory) and improved UNet. In the hybrid network, improved UNet is utilized to extract spatial features. Then Bi-LSTM cells are applied to extract temporal features and memorize the change of sequential data in forward and backward directions. The proposed method is applied to the well-known 5 public databases and has given outperforming results than the current state of the art.

Polyphonic piano transcription based on graph convolutional network

The automatic music transcription (AMT) task is designed to convert raw performance audio signals into digital representations of symbolic music for possible computational musicology. In polyphonic music, there are multiple notes and they may appear at the same time. The combined output of multiple notes faces the problem of dimension explosion, which makes it difficult to achieve accurate transcription. To overcome the above challenge, a deep learning model based on graph convolution network (CR-GCN) is proposed to cope with the problem of dimension explosion by exploring the interdependence between musical notes. The model is divided into two parts, i.e., feature learning and label learning. Feature learning is composed of serial convolutional neural network (CNN) and recurrent neural network (RNN), which aims to extract temporal and spatial features from the input music signals. Label learning consists of graph convolutional network (GCN), which is adopted to model the interdependence between notes. The CR-GCN model can be end-to-end trainable through joint training feature networks and label networks. Experiments on public polyphonic music data sets show that the proposed method is able to mine more co-existing notes, and it is superior to existing methods in both frame-level and note-level indexes. Moreover, visual analysis shows that the learned interdependence between notes has good explainability in music theory.

Multi-Task Learning Radar Transformer (MLRT): A Personal Identification and Fall Detection Network Based on IR-UWB Radar.

Radar-based personal identification and fall detection have received considerable attention in smart healthcare scenarios. Deep learning algorithms have been introduced to improve the performance of non-contact radar sensing applications. However, the original Transformer network is not suitable for multi-task radar-based applications to effectively extract temporal features from time-series radar signals. This article proposes the Multi-task Learning Radar Transformer (MLRT): a personal Identification and fall detection network based on IR-UWB radar. The proposed MLRT utilizes the attention mechanism of Transformer as its core to automatically extract features for personal identification and fall detection from radar time-series signals. Multi-task learning is applied to exploit the correlation between the personal identification task and the fall detection task, enhancing the performance of discrimination for both tasks. In order to suppress the impact of noise and interference, a signal processing approach is employed including DC removal and bandpass filtering, followed by clutter suppression using a RA method and Kalman filter-based trajectory estimation. An indoor radar signal dataset is generated with 11 persons under one IR-UWB radar, and the performance of MLRT is evaluated using this dataset. The measurement results show that the accuracy of MLRT improves by 8.5% and 3.6% for personal identification and fall detection, respectively, compared to state-of-the-art algorithms. The indoor radar signal dataset and the proposed MLRT source code are publicly available.

A Novel Spiking Graph Attention Network for Intelligent Fault Diagnosis of Planetary Gearboxes

As an essential component in the power transmission system of rotating machinery, planetary gearboxes are often subjected to extreme operating conditions, including heavy loads and low speeds, which can cause key components to fail over time. Early planetary gearbox fault features are weak, coupled with the impact of strong noise in the industrial field, which brings great challenges to the accurate identification and diagnosis of planetary gearbox faults. To address these issues, we propose a novel spiking graph attention network (Spiking-GAT) for intelligent fault diagnosis of planetary gearboxes, which realizes synchronous extraction of temporal and spatial features of the original signal, to accurately identify the fault category and severity of planetary gearbox. Firstly, a graph data construction method based on chaos theory is proposed. A multi-phase coupled chaotic oscillator array based on Duffing (MP-COAD) is established to reconstruct the original signal, and the K-nearest neighbor classifier (KNN) is used to construct the reconstructed signal as graph data. Secondly, a novel Spiking GAT intelligent fault diagnosis framework has been developed, which provides an adaptive spiking coding approach for graph data and deep mining and extraction of spatial-temporal features to accurately identify the health status of planetary gearboxes. Finally, the experimental and industrial fault simulation test rigs of the planetary gearbox are designed, and three cases are studied to verify the accuracy and stability of Spiking-GAT. The results show that the classification accuracy of Spiking-GAT reaches 100% on the datasets of the two test rigs, and has excellent noise robustness.

Classification of EEG signals using Transformer based deep learning and ensemble models

A Brain-Computer Interface (BCI) is a communication and control system designed to provide interaction between a user and a computer device. This interaction is based on the brain's electrical signals that are generated when users do specific tasks. Different categories of visual stimuli evoke distinct activation patterns in the human brain. The generated patterns can be recorded with EEG signals for use in BCI applications. Recently, deep learning-based Transformer models have demonstrated significant potential for analyzing diverse data. In this paper, a new Transformer-based model has been presented that extracts temporal and spectral features from EEG signals for classification purposes. The proposed Spectral Transformer model converts the EEG signal to the frequency domain using PSD before applying Transformer models to extract frequency features. Deep ensemble learning models are used to enhance the generalization performance of the final model by combining the benefits of both deep learning models and ensemble learning. The proposed ensemble model combines Temporal and Spectral Transformers to simultaneously utilize the time and frequency features of the signal. The accuracy of 96.1 %, 94.20 %, and 93.60 % are achieved using an ensemble model, Temporal Transformer, and Spectral Transformer, respectively. These results demonstrate the effectiveness of the proposed model for accurately classifying EEG signals in BCI applications.

TFC-GCN: Lightweight Temporal Feature Cross-Extraction Graph Convolutional Network for Skeleton-Based Action Recognition

For skeleton-based action recognition, graph convolutional networks (GCN) have absolute advantages. Existing state-of-the-art (SOTA) methods tended to focus on extracting and identifying features from all bones and joints. However, they ignored many new input features which could be discovered. Moreover, many GCN-based action recognition models did not pay sufficient attention to the extraction of temporal features. In addition, most models had swollen structures due to too many parameters. In order to solve the problems mentioned above, a temporal feature cross-extraction graph convolutional network (TFC-GCN) is proposed, which has a small number of parameters. Firstly, we propose the feature extraction strategy of the relative displacements of joints, which is fitted for the relative displacement between its previous and subsequent frames. Then, TFC-GCN uses a temporal feature cross-extraction block with gated information filtering to excavate high-level representations for human actions. Finally, we propose a stitching spatial–temporal attention (SST-Att) block for different joints to be given different weights so as to obtain favorable results for classification. FLOPs and the number of parameters of TFC-GCN reach 1.90 G and 0.18 M, respectively. The superiority has been verified on three large-scale public datasets, namely NTU RGB + D60, NTU RGB + D120 and UAV-Human.

Omni-Domain Feature Extraction Method for Gait Recognition

As a biological feature with strong spatio-temporal correlation, the current difficulty of gait recognition lies in the interference of covariates (viewpoint, clothing, etc.) in feature extraction. In order to weaken the influence of extrinsic variable changes, we propose an interval frame sampling method to capture more information about joint dynamic changes, and an Omni-Domain Feature Extraction Network. The Omni-Domain Feature Extraction Network consists of three main modules: (1) Temporal-Sensitive Feature Extractor: injects key gait temporal information into shallow spatial features to improve spatio-temporal correlation. (2) Dynamic Motion Capture: extracts temporal features of different motion and assign weights adaptively. (3) Omni-Domain Feature Balance Module: balances fine-grained spatio-temporal features, highlight decisive spatio-temporal features. Extensive experiments were conducted on two commonly used public gait datasets, showing that our method has good performance and generalization ability. In CASIA-B, we achieved an average rank-1 accuracy of 94.2% under three walking conditions. In OU-MVLP, we achieved a rank-1 accuracy of 90.5%.

Dynamic prediction for attitude and position of shield machine in tunneling: A hybrid deep learning method considering dual attention

In constructing long-distance shield tunnels, it is a difficult challenge to maintain the tunneling trajectory consistent with the design tunnel axis. The accurate prediction of the attitude and position during tunneling can reap the advantage of optimizing the tunneling operation parameters in advance leading to the best tunneling trajectory. This study investigates a framework based on a hybrid deep learning model for attitude and position prediction of the shield machine. This prediction framework contains comprehensive feature evaluation method, ensemble empirical mode decomposition (EEMD), convolutional neural network (CNN), and gate recurrent unit (GRU). The introduction of channel attention and temporal attention in CNN and GRU further strengthens the spatial and temporal feature extraction ability of the model. The performance of the prediction framework is verified through a case study with data collected from the Shanghai urban railway tunnel section. Results reveal that the proposed model with dual attention significantly outperforms other models in prediction accuracy and speed. The bias of feature data can be alleviated by introducing channel attention, and using temporal attention can capture long-distance temporal feature data. The model can support shield construction safety by adjusting the operation parameters, and an application example is used to demonstrate the feasibility of the proposed approach.

Information diffusion prediction based on cascade sequences and social topology

People have been increasingly sharing information via social media platforms such as Twitter and Facebook in recent years, making understanding and predicting the spread of information a hot research topic. Accurate prediction of cascades within social platforms can effectively track the information dissemination process and prevent the spread of harmful information. Previous work either did not fully exploit the potential social graph structure or chose to predict communication sequences as a diffusion graph, resulting in inconsistency with the actual diffusion situation. We propose a method that is structurally simple and effective, requiring only sequential sequences of users' spread information and social relationships among users. Our method is based on LSTM (Long Short-Term Memory) for temporal feature extraction of cascade sequences and combined with GCN (Graph Convolutional Networks) for extracting features containing node topology in social graphs for microscopic prediction of information diffusion cascades. Comparative experiments on two real social platform datasets demonstrate that our method has a better prediction effect.

Sequential sparse autoencoder for dynamic heading representation in ventral intraparietal area

To navigate in space, it is important to predict headings in real-time from neural responses in the brain to vestibular and visual signals, and the ventral intraparietal area (VIP) is one of the critical brain areas. However, it remains unexplored in the population level how the heading perception is represented in VIP. And there are no commonly used methods suitable for decoding the headings from the population responses in VIP, given the large spatiotemporal dynamics and heterogeneity in the neural responses. Here, responses were recorded from 210 VIP neurons in three rhesus monkeys when they were performing a heading perception task. And by specifically and separately modelling the both dynamics with sparse representation, we built a sequential sparse autoencoder (SSAE) to do the population decoding on the recorded dataset and tried to maximize the decoding performance. The SSAE relies on a three-layer sparse autoencoder to extract temporal and spatial heading features in the dataset via unsupervised learning, and a softmax classifier to decode the headings. Compared with other population decoding methods, the SSAE achieves a leading accuracy of 96.8% ± 2.1%, and shows the advantages of robustness, low storage and computing burden for real-time prediction. Therefore, our SSAE model performs well in learning neurobiologically plausible features comprising dynamic navigational information.

Spatial–temporal features-based EEG emotion recognition using graph convolution network and long short-term memory

Objective. Emotion recognition on the basis of electroencephalography (EEG) signals has received a significant amount of attention in the areas of cognitive science and human–computer interaction (HCI). However, most existing studies either focus on one-dimensional EEG data, ignoring the relationship between channels, or only extract time–frequency features while not involving spatial features. Approach. We develop spatial–temporal features-based EEG emotion recognition using a graph convolution network (GCN) and long short-term memory (LSTM), named ERGL. First, the one-dimensional EEG vector is converted into a two-dimensional mesh matrix, so that the matrix configuration corresponds to the distribution of brain regions at EEG electrode locations, thus to represent the spatial correlation between multiple adjacent channels in a better way. Second, the GCN and LSTM are employed together to extract spatial–temporal features; the GCN is used to extract spatial features, while LSTM units are applied to extract temporal features. Finally, a softmax layer is applied to emotion classification. Main results. Extensive experiments are conducted on the A Dataset for Emotion Analysis using Physiological Signals (DEAP) and the SJTU Emotion EEG Dataset (SEED). The classification results of accuracy, precision, and F-score for valence and arousal dimensions on DEAP achieved 90.67% and 90.33%, 92.38% and 91.72%, and 91.34% and 90.86%, respectively. The accuracy, precision, and F-score of positive, neutral, and negative classifications reached 94.92%, 95.34%, and 94.17%, respectively, on the SEED dataset. Significance. The above results demonstrate that the proposed ERGL method is encouraging in comparison to state-of-the-art recognition research.

The Human Continuity Activity Semisupervised Recognizing Model for Multiview IoT Network

With advances in spatial-temporal internet of things (IoT) technologies, human activity recognition (HAR) has played a major role in human safety and medical health. Recently, most works focus on activity deep feature extraction, offering promising alternatives to manually engineered features. However, how to extract the effective and distinguishable continuity activity features and meanwhile reduce the heavy dependence on labels still remains the key challenge for human activity recognition. This paper proposes the human continuity activity semi-supervised recognizing method in multi-view IoT network scenarios. Our innovation combines supervised activity feature extraction with unsupervised encoder-decoder modules, which can capture continuity activity features from sensor data streams. To be more specific, our work applies convolutional neural network (CNN) to capture the local dependence of sensor data and designs an encoder-decoder architecture to extract temporal features in an unsupervised manner. Then we fuse these two features to recognize activities and train them with manual labels, thereby refining the temporal feature extraction and training CNN module. Experiments on five public datasets demonstrate the exceptional performance of our proposed method, which can achieve a higher recognition accuracy on almost all the datasets and is more robust and adaptive among different datasets.

The LST-SATM-net: A new deep feature learning framework for aero-engine hydraulic pipeline systems intelligent faults diagnosis

Accurate identification of faults in the aero-engine hydraulic pipelines is of great significance for condition monitoring of engines as well as safe and reliable operation of aircraft. However, due to the complexity of the structure of the aero-engine pipeline systems, the correlation of the coupling, and the multi-source nature of the excitation it is subjected to, the identification of failure characteristics has been an urgent problem in both engineering and academia. In addition, so there is no more qualitative formula or characteristic index to describe the fault of the aero-hydraulic pipeline. Therefore, to overcome these challenges, in this paper, a new compound faults diagnosis framework, namely Lightweight Spatial-Temporal Model Fusion Self-Attention Mechanism (LST-SATM-Net), is proposed for the aero-engine hydraulic pipeline systems. Firstly, the lightweight spatial feature extraction module is designed to extract fine-grained features from aero-engine hydraulic pipeline fault data and the bi-directional temporal feature extraction module is designed to extract coarse-grained features with temporal patterns from fine-grained features, achieving coarse and fine-grained feature fusion. Then, a self-attention mechanism is integrated into the spatial–temporal model for optimization, enabling the model to automatically learn important features from data with multiple sources of coupled vibration excitation, enhancing the performance of critical features from the correlated coupling of compound faults. The powerful noise interference is suppressed and the weighting coefficients are continuously updated to make the final decision more focused. Consequently, the recognition accuracy of the LST-SATM-Net model is improved and the feature learning of the neural network model is more flexible. The results demonstrated that compared with seven other advanced methods, the proposed LST-SATM-Net model is capable of diagnosing compound faults in the hydraulic pipeline system of aero-engine more accurately after being trained by experimental measurement data. This impressive finding indicates that the proposed LST-SATM-Net model can be applied to the actual condition monitoring of aero-engine hydraulic pipelines to reduce maintenance costs and downtime.

Memory long and short term time series network for ultra-short-term photovoltaic power forecasting

Photovoltaic (PV) power is stochastic, intermittent and volatile, which has brought huge challenges to the safe and stable operation of the power grid. Accurate PV power forecasting is becoming a significant task for PV plant grid connecting, scheduling and guaranteeing the safety of the power grid. To improve the accuracy of PV power forecasting, a memory long and short term time series network (MLSTNet) model is proposed to perform ultra-short-term PV power forecasting with a time horizon varying from 15 min to 4 h. Firstly, the input variables are screened by calculating Spearman correlation coefficients between weather data. To cluster different types of data, the classification coefficient is set. Secondly, different type data are automatically identified in the MLSTNet model. The appropriate model parameters and network structures are adjusted for the higher forecasting speed and accuracy. By using the single-step rolling input method and the temporal attention convolutional neural network, the ability of temporal and spatial feature extraction has been significantly enhanced, allowing real-time forecasting and dynamic optimization of the proposed model. Thirdly, long-term and short-term historical data are fed into the model, and different periods dependencies are captured while incorporating multiple variable dimensions, leading to more memorable in the network. Finally, the experimental data is selected from the actual operation data of a PV plant in northern China, which represents the universal applicability and practical application of the model. The performance of the proposed model is demonstrated with actual PV plant dataset, the RMSE of proposed model decreased by 32.73%, 5.44%, 12.44%, and 31.22%. Results indicated that it is feasible to use the proposed MLSTNet model for time series forecasting. The core idea of the MLSTNet model is to achieve higher accuracy in ultra-short-term forecasting by enhancing the ability to extract temporal features and linearity in the model, providing a new idea for deep learning-based methods for ultra-short-term PV power forecasting.