Temporal Convolutional Network Research Articles

Revealing Urban Deformation Patterns through InSAR Time Series Analysis with TCN and Transfer Learning

Abstract. Current multi-epoch InSAR techniques heavily rely on the assumption of linear deformation. This can sometimes overlook crucial deformation signals when using velocities for evaluation. The process of interpreting InSAR time series is not only time-consuming and labor-intensive but also requires a certain level of expertise. This study refines existing InSAR deformation categories, such as stable, linear, step, piecewise linear, power, and undefined, to define 'canonical deformation time series patterns.' We propose an innovative approach for InSAR post-processing using Temporal Convolutional Networks (TCN) and transfer learning. Due to the limited availability of real data, we use simulated data to train a pre-existing model. We then assess the effectiveness of our method in identifying urban deformation patterns. This research could significantly improve our understanding of large-scale InSAR time series deformation and reveal the underlying patterns.

Short-term power load forecasting for integrated energy system based on a residual and attentive LSTM-TCN hybrid network

In the context of Integrated Energy System (IES), accurate short-term power demand forecasting is crucial for ensuring system reliability, optimizing operational efficiency through resource allocation, and supporting effective real-time decision-making in energy management. However, achieving high forecasting accuracy faces significant challenges due to the inherent complexity and stochastic nature of IES’s short-term load profiles, resulting from diverse consumption patterns among end-users and the intricate coupling within the network of interconnected energy sources. To address this issue, a dedicated Short-Term Power Load Forecasting (STPLF) framework for IES is proposed, which relies on a newly developed hybrid deep learning architecture. The framework seamlessly combines Long Short-Term Memory (LSTM) with Temporal Convolutional Network (TCN), enhanced by an attention mechanism module. By merging these methodologies, the network leverages the parallel processing prowess of TCN alongside LSTM’s ability to retain long-range temporal information, thus enabling it to dynamically concentrate on relevant sections of time series data. This synergy leads to improved prediction accuracy and broader applicability. Furthermore, the integration of residual connections within the network structure serves to deepen its learning capabilities and enhance overall performance. Ultimately, results from a real case study of a user-level IES demonstrate that the Mean Absolute Percentage Error (MAPE) of the proposed framework on the test set is 2.35%. This error rate is lower than the averages of traditional methods (3.43%) and uncombined single submodules (2.80%).

Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition

AbstractContinuous emotion recognition is to predict emotion states through affective information and more focus on the continuous variation of emotion. Fusion of electroencephalography (EEG) and facial expressions videos has been used in this field, while there are with some limitations in current researches, such as hand‐engineered features, simple approaches to integration. Hence, a new continuous emotion recognition model is proposed based on the fusion of EEG and facial expressions videos named residual multimodal Transformer (RMMT). Firstly, the Resnet50 and temporal convolutional network (TCN) are utilised to extract spatiotemporal features from videos, and the TCN is also applied to process the computed EEG frequency power to acquire spatiotemporal features of EEG. Then, a multimodal Transformer is used to fuse the spatiotemporal features from the two modalities. Furthermore, a residual connection is introduced to fuse shallow features with deep features which is verified to be effective for continuous emotion recognition through experiments. Inspired by knowledge distillation, the authors incorporate feature‐level loss into the loss function to further enhance the network performance. Experimental results show that the RMMT reaches a superior performance over other methods for the MAHNOB‐HCI dataset. Ablation studies on the residual connection and loss function in the RMMT demonstrate that both of them is functional.

Incorporating the Third Law of Geography with Spatial Attention Module–Convolutional Neural Network–Transformer for Fine-Grained Non-Stationary Air Quality Predictive Learning

Accurate air quality prediction is paramount in safeguarding public health and addressing air pollution control. However, previous studies often ignore the geographic similarity among different monitoring stations and face challenges in dynamically capturing different spatial–temporal relationships between stations. To address this, an air quality predictive learning approach incorporating the Third Law of Geography with SAM–CNN–Transformer is proposed. Firstly, the Third Law of Geography is incorporated to fully consider the geographical similarity among stations via a variogram and spatial clustering. Subsequently, a spatial–temporal attention convolutional network that combines the spatial attention module (SAM) with the convolutional neural network (CNN) and Transformer is designed. The SAM is employed to extract spatial–temporal features from the input data. The CNN is utilized to capture local information and relationships among each input feature. The Transformer is applied to capture time dependencies across long-distance time series. Finally, Shapley’s analysis is employed to interpret the model factors. Numerous experiments with two typical air pollutants (PM2.5, PM10) in Haikou City show that the proposed approach has better comprehensive performance than baseline models. The proposed approach offers an effective and practical methodology for fine-grained non-stationary air quality predictive learning.

Enhancing disruption prediction through Bayesian neural network in KSTAR

Abstract In this research, we develop a data-driven disruption predictor based on Bayesian deep probabilistic learning, capable of predicting disruptions and modeling uncertainty in KSTAR. Unlike conventional neural networks within a frequentist approach, Bayesian neural networks can quantify the uncertainty associated with their predictions, thereby enhancing the precision of disruption prediction by mitigating false alarm rates through uncertainty thresholding. Leveraging 0D plasma parameters from EFIT and diagnostic data, a Temporal Convolutional Network adept at handling multi-time scale data was utilized. The proposed framework demonstrates proficiency in predicting disruptions, substantiating its effectiveness through successful applications to KSTAR experimental data.

Geographic heterogeneity of activation functions in urban real-time flood forecasting: Based on seasonal trend decomposition using Loess-Temporal Convolutional Network-Gated Recurrent Unit model

Urban real-time flood forecasting is crucial for flood prevention and sustainable development, but it poses challenges due to data inputs and activation functions selection in data-driven models without sufficient focus of geographic heterogeneity. In this study, a novel Seasonal Trend Decomposition using Loess (STL)-Temporal Convolutional Network (TCN)-Gated Recurrent Unit (GRU) model is proposed to improve urban real-time flood forecasting accuracy, twenty-one different activation functions are considered for geographic heterogeneity. Experiments are conducted at six urban drainage system locations in Odense, Denmark. The results show that: (1) STL effectively prepares data for forecasting using TCN and GRU models, leading improved performance compared to single models. STL-TCN-GRU deep learning model demonstrates strong applicability in urban forecasting, achieving an overall accuracy of 0.0079, 0.0140, 0.0482 and 0.9581 in Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE) and Nash-Sutcliffe efficiency coefficient (NSE), respectively. (2) Softsign emerges as the best activation function for forecasting in lower drainage system locations, with average accuracy of 0.0094, 0.0018, 0.0049 and 0.9938 in MAE, RMSE, MAPE and NSE, respectively. Furthermore, Softsign proves to be the best activation function for forecasting in middle drainage system locations, with average accuracy of 0.0047, 0.0081, 0.0309 and 0.9547 in MAE, RMSE, MAPE and NSE, respectively. Swish is the best activation function for forecasting in upper drainage system locations, with average accuracy of 0.0052, 0.0080, 0.0627 and 0.9060 in MAE, RMSE, MAPE and NSE, respectively. This study provides valuable insights for urban real-time flood forecasting modeling with high accuracy and evidence for activation function selection in data-driven models like STL-TCN-GRU for geographic heterogeneity.

A Novel Hybrid Deep Learning Model for Complex Systems: A Case of Train Delay Prediction

Predicting the status of train delays, a complex and dynamic problem, is crucial for railway enterprises and passengers. This paper proposes a novel hybrid deep learning model composed of convolutional neural networks (CNN) and temporal convolutional networks (TCN), named the CNN + TCN model, for predicting train delays in railway systems. First, we construct 3D data containing the spatiotemporal characteristics of real-world train data. Then, the CNN + TCN model employs a 3D CNN component, which is fed into the constructed 3D data to mine the spatiotemporal characteristics, and a TCN component that captures the temporal characteristics in railway operation data. Furthermore, the characteristic variables corresponding to the two components are selected. Finally, the model is evaluated by leveraging data from two railway lines in the United Kingdom. Numerical results show that the CNN + TCN model has greater accuracy and convergence performance in train delay prediction.

DTT: A Dual-domain Transformer model for Network Intrusion Detection

With the rapid evolution of network technologies, network attacks have become increasingly intricate and threatening. The escalating frequency of network intrusions has exerted a profound influence on both industrial settings and everyday activities. This underscores the urgent necessity for robust methods to detect malicious network traffic. While intrusion detection techniques employing Temporal Convolutional Networks (TCN) and Transformer architectures have exhibited commendable classification efficacy, most are confined to the temporal domain. These methods frequently fall short of encompassing the entirety of the frequency spectrum inherent in network data, thereby resulting in information loss. To mitigate this constraint, we present DTT, a novel dual-domain intrusion detection model that amalgamates TCN and Transformer architectures. DTT adeptly captures both high-frequency and low-frequency information, thereby facilitating the simultaneous extraction of local and global features. Specifically, we introduce a dual-domain feature extraction (DFE) block within the model. This block effectively extracts global frequency information and local temporal features through distinct branches, ensuring a comprehensive representation of the data. Moreover, we introduce an input encoding mechanism to transform the input into a format suitable for model training. Experiments conducted on two distinct datasets address concerns regarding data duplication and diverse attack types, respectively. Comparative experiments with recent intrusion detection models unequivocally demonstrate the superior performance of the proposed DTT model.

Interpretable and explainable hybrid model for daily streamflow prediction based on multi-factor drivers.

Streamflow time series data typically exhibit nonlinear and nonstationary characteristics that complicate precise estimation. Recently, multifactorial machine learning (ML) models have been developed to enhance the performance of streamflow predictions. However, the lack of interpretability within these ML models raises concerns about their inner workings and reliability. This paper introduces an innovative hybrid architecture, the TCN-LSTM-Multihead-Attention model, which combines two layers of temporal convolutional networks (TCN) followed by one layer of long short-term memory (LSTM) units, integrated with a Multihead-Attention mechanism for predicting streamflow with streamflow causation-driven prediction samples (RCDP), employing local and global interpretability studies through Shapley values and partial dependency analysis. The find_peaks method was used to identify peak flow events in the test dataset, validating the model's generality and uncovering the physical causative patterns of streamflow. The results show that (1) compared to the LSTM model with the same hyperparameter settings, the proposed TCN-LSTM-Multihead-Attention hybrid model increased the R2 by 52.9%, 2.5%, 43.1%, and 10.7% respectively at four stations in the test set predictions using RCDP samples. Moreover, comparing the prediction results of the hybrid model under different samples in Hengshan station, the R2 for RCDP increased by 5.06% and 1.22% compared to streamflow autoregressive prediction samples (RAP) and meteorological-soil volumetric water content coupled autoregressive prediction samples (MCSAP) respectively. (2) Historical streamflow data from the preceding 3days predominantly influences predictions due to strong autocorrelation, with flow quantity (Q) typically emerging as the most significant feature alongside precipitation (P), surface soil moisture (SSM), and adjacent station flow data. (3) During periods of low and normal flow, historical data remains the most crucial factor; however, during flood periods, the roles of upstream inflow and precipitation become significantly more pronounced. This model facilitates the identification and quantification of various hydrodynamic impacts on flow predictions, including upstream flood propagation, precipitation, and soil moisture conditions. It also elucidates the model's nonlinear relationships and threshold responses, thereby enhancing the interpretability and reliability of streamflow predictions.

MTTLA‐DLW: Multi‐task TCN‐Bi‐LSTM transfer learning approach with dynamic loss weights based on feature correlations of the training samples for short‐term wind power prediction

AbstractWind power prediction for newly built wind farms is usually faced with the problem of no sufficient historical data. To efficiently extract the useful features from related wind farms, a novel transfer learning method based on temporal convolutional network (TCN)‐Bi‐long short‐term memory (LSTM) with dynamic loss weights is proposed. Firstly, a novel multi‐task TCN‐Bi‐LSTM model is designed to extract common features. The separate TCNs, and common Bi‐LSTM layers of the proposed model are designed to extract the temporal features from related wind farms. Secondly, in the pre‐training stage, to optimize the training process of the neural networks, a dynamic loss‐weighting strategy is proposed for multi‐task learning (MTL) to select the most related features, which increase the prediction accuracy by providing a suitable optimization object. Thirdly, the multi‐task TCN‐Bi‐LSTM model is re‐trained based on the samples from the target wind farm. Finally, a dataset of seven wind farms was employed to evaluate the efficiency of the proposed MTL structure and the dynamic loss‐weighting strategy. The result shows that the root mean squared error of the 12‐h short‐term prediction can be decreased by 4.19% compared with the traditional single‐task learning model, which verifies the validity of the proposed multi‐task transfer learning method.

Fusion inception and transformer network for continuous estimation of finger kinematics from surface electromyography

Decoding surface electromyography (sEMG) to recognize human movement intentions enables us to achieve stable, natural and consistent control in the field of human computer interaction (HCI). In this paper, we present a novel deep learning (DL) model, named fusion inception and transformer network (FIT), which effectively models both local and global information on sequence data by fully leveraging the capabilities of Inception and Transformer networks. In the publicly available Ninapro dataset, we selected surface EMG signals from six typical hand grasping maneuvers in 10 subjects for predicting the values of the 10 most important joint angles in the hand. Our model’s performance, assessed through Pearson’s correlation coefficient (PCC), root mean square error (RMSE), and R-squared (R2) metrics, was compared with temporal convolutional network (TCN), long short-term memory network (LSTM), and bidirectional encoder representation from transformers model (BERT). Additionally, we also calculate the training time and the inference time of the models. The results show that FIT is the most performant, with excellent estimation accuracy and low computational cost. Our model contributes to the development of HCI technology and has significant practical value.

Integrating deep transformer and temporal convolutional networks for SMEs revenue and employment growth prediction

Although the accurate potential for growth prediction is very important for Government grants and contributions programs to better support Small and Medium-sized Enterprises (SMEs), it is a challenging task due to the data heterogeneity (both structured data and free text data bilingual in English and French), the class imbalance issue, and the difficulties in efficient feature learning. To address these challenges, this paper presents a novel BERT-TCN model for portfolio predictions in government funding programs, with the following key contributions. First, we describe the application of a novel architecture to a prediction task involving sequential, structured, partially quantitative input data and free text input data. Specifically, our novel model predicts the growth of firms receiving government funding for innovation. Our model also deals with class imbalance in the data and the difficulties in efficient feature learning. Our model integrates a Transformer model, i.e., BERT, for text modeling with a Temporal Convolutional Network (TCN) for sequential prediction. Second, we also developed various performance evaluation criteria in Section 4.3, allowing comprehensive assessments of the proposed approach from both the machine learning perspective and funding program-specific perspective. Third, the importance of features (both text and numerical features) is quantified and evaluated, allowing insights into how different features contribute to the prediction and explainability of the proposed model. The proposed approach is trained and tested on a large dataset from a rich database, demonstrating that the proposed approach can greatly help individual human experts improve their results.

Unsupervised anomaly detection for power batteries: A temporal convolution autoencoder framework

Abstract To prevent potential abnormalities from escalating into critical faults, a rapid and precise algorithm should be employed for detecting power battery anomalies. An unsupervised model based on temporal convolutional autoencoder (TCAE) that can quickly and accurately identify abnormal power battery data was proposed. Its encoder utilized a temporal convolutional network (TCN) structure with residuals to parallelly process data while capturing time dependencies. A novel TCN structure with an effect–cause relationship was developed for the decoder. The same-time-scale connection was established between the encoder and decoder to improve the model performance. The validity of the proposed model was confirmed using a real-world car dataset. Compared with the GRU-AE model, the proposed approach reduced the parameter count and mean square error by 19.5% and 71.9%, respectively. This study provides insights into the intelligent battery pack abnormality detection technology.

Prediction of Icing on Wind Turbines Based on SCADA Data via Temporal Convolutional Network

Icing on the blades of wind turbines during winter seasons causes a reduction in power and revenue losses. The prediction of icing before it occurs has the potential to enable mitigating actions to reduce ice accumulation. This paper presents a framework for the prediction of icing on wind turbines based on Supervisory Control and Data Acquisition (SCADA) data without requiring the installation of any additional icing sensors on the turbines. A Temporal Convolutional Network is considered as the model to predict icing from the SCADA data time series. All aspects of the icing prediction framework are described, including the necessary data preprocessing, the labeling of SCADA data for icing conditions, the selection of informative icing features or variables in SCADA data, and the design of a Temporal Convolutional Network as the prediction model. Two performance metrics to evaluate the prediction outcome are presented. Using SCADA data from an actual wind turbine, the model achieves an average prediction accuracy of 77.6% for future times of up to 48 h.

Lithium-ion battery SOH prediction based on VMD-PE and improved DBO optimized temporal convolutional network model

Accurately estimating the state of health (SOH) in lithium-ion batteries is crucial for optimizing performance, cost management, enhancing safety, extending lifespan, and promoting environmental sustainability. However, due to factors such as capacity regeneration, battery capacity degradation data exhibits nonlinear and non-stationary characteristics, making it challenging to accurately predict SOH. In order to improve prediction accuracy, this paper proposes a novel SOH prediction model based on VMD-PE-IDBO-TCN. Firstly, the variational mode decomposition-permutation entropy (VMD-PE) method is innovatively introduced to decompose the original battery SOH data and reorganize it into a series of subsequences, reducing the impact of non-stationary components on the prediction results. Moreover, temporal convolutional network (TCN) prediction models are used for each subsequence, and an improved dung beetle optimization (IDBO) algorithm is used to optimize the TCN parameters, where this IDBO algorithm incorporates SPM chaotic mapping, golden sine strategy, and adaptive Gaussian-Cauchy mixed mutation perturbation. The IDBO algorithm rapidly and accurately identifies the optimal parameter combination, significantly enhancing the predictive performance of the TCN model. Finally, the predicted values of each subsequence are superimposed to obtain the final prediction result. Two different lithium-ion battery datasets from NASA and CALCE are used for experimental validation. Experimental results show that the proposed model has higher prediction accuracy, with a maximum RMSE of only 0.0072 and a maximum MAPE of only 0.67 %, and has good adaptability to different types of batteries and batteries operating at high temperatures.

Classification of Gaits With a High Risk of Falling Using a Head-Mounted Device With a Temporal Convolutional Network

Classification of Gaits With a High Risk of Falling Using a Head-Mounted Device With a Temporal Convolutional Network

A short-term wind power prediction approach based on an improved dung beetle optimizer algorithm, variational modal decomposition, and deep learning

Accurate short-term wind power prediction is crucial for the efficient and safe operation of wind power systems. To enhance the accuracy of short-term wind power prediction, this paper proposes a hybrid short-term wind power prediction model, IDBO-VMD-TCN-GRU-Attention, based on the Improved Dung Beetle Optimizer algorithm (IDBO), Variational Modal Decomposition (VMD), Temporal Convolutional Network (TCN), Gated Recurrent Unit (GRU), and Attention Mechanism. The IDBO is used to optimize the parameters of the VMD. Then the optimized IDBO-VMD is used to decompose the original data into modal components with lower volatility to fully extract the data features. These modal components are inputted into the TCN-GRU-Attention to predict the short-term wind power and obtain the final prediction value. The model is tested and validated using real data from four different months and compared against 11 other models. Results demonstrate that our proposed model significantly reduces the Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) by an average of at least 22.86 %, 18.82 %, and 19.99 %, respectively, across the four different months. This indicates that the proposed model provides a superior fit to the data, offers higher prediction accuracy, and holds significant practical value.

A brain topography graph embedded convolutional neural network for EEG-based motor imagery classification

Brain-computer interface (BCI) based on motor imagery EEG (MI-EEG) has been used extensively in health care, device control, entertainment, and other fields. MI-EEG signals consist of multiple channels that exhibit a specific topological relationship. However, many existing methods based on deep learning do not make good use of the multi-electrode characteristics of EEG signals. To further improve the decoding performance of MI-EEG, this study proposes a novel method combining the brain topology graph embedding and convolutional neural network (CNN) for MI-EEG classification. Specifically, mutual information is used to measure the relationship between EEG channels. A graph convolutional network (GCN) is employed to embed the topological relationship. Following this, a convolutional block using depth-wise convolution is built to extract spatial–temporal features, while a temporal convolutional network (TCN) is used to extract advanced temporal features. The model’s performance is evaluated on BCICIV-2a and HGD datasets. The experimental results show that the classification accuracy of the proposed model is 80.46% and 94.38%, respectively, outperforming existing methods. Moreover, ablation experiments and feature visualization show that graph embedding can significantly improve the decoding performance of MI-EEG.

Prediction of Parking Space Availability Using Improved MAT-LSTM Network

Prediction of Parking Space Availability Using Improved MAT-LSTM Network

A channel-fused gated temporal convolutional network for EMG-based gesture recognition

Background:EMG gesture recognition can be widely applied in many fields, such as prosthetic control and human–computer interaction, to enhance the standard of human life. Purpose:This study aims to develop a deep learning-based model to identify multiple complex gestures from raw EMG signals. Methods:We propose a new channel-fused gated temporal convolutional network. First, a channel fusion and gating mechanism is designed to improve temporal convolutional networks, allowing the model to obtain higher-level features. Second, we improve the channel fusion module by the short-term average energy to fuse the EMG signals of multi-channels more accurately. Results:The model is evaluated on a public dataset of EMG gestures, NinaPro DB5. Results demonstrated that the unbalanced accuracy rate of 53 gesture actions reaches 92.71%, and the balanced accuracy rate 74.79%. Further, ablation experiments validates the effectiveness of each model module. Conclusions:This study demonstrates our proposed approach can improve gesture recognition accuracy for complex gestures and has great potential for practical applications.