Temporal Convolutional Neural Network Research Articles

Investigating Personalization Techniques for Improved Cybersickness Prediction in Virtual Reality Environments.

In recent cybersickness research, there has been a growing interest in predicting cybersickness using real-time physiological data such as heart rate, galvanic skin response, eye tracking, postural sway, and electroencephalogram. However, the impact of individual factors such as age and gender, which are pivotal in determining cybersickness susceptibility, remains unknown in predictive models. Our research seeks to address this gap, underscoring the necessity for a more personalized approach to cybersickness prediction to ensure a better, more inclusive virtual reality experience. We hypothesize that a personalized cybersickness prediction model would outperform non-personalized models in predicting cybersickness. Evaluating this, we explored four personalization techniques: 1) data grouping, 2) transfer learning, 3) early shaping, and 4) sample weighing using an open-source cybersickness dataset. Our empirical results indicate that personalized models significantly improve prediction accuracy. For instance, with early shaping, the Deep Temporal Convolutional Neural Network (DeepTCN) model achieved a 69.7% reduction in RMSE compared to its non-personalized version. Our study provides evidence of personalization techniques' benefits in improving cybersickness prediction. These findings have implications for developing personalized cybersickness prediction models tailored to individual differences, which can be used to develop personalized cybersickness reduction techniques in the future.

Estimation of the pavement skid resistance by automated vehicle based on one-dimensional convolutional neural network

ABSTRACT The pavement skid resistance is critical to the driving safety of a road. Automated vehicles, which have high requirements for environmental perception, are not yet commonly equipped with an estimation module that could monitor the real-time skid resistance, leading thus to instability in the steering and braking control. In this paper, we propose a data-driven approach incorporating vehicle dynamics implementing real-time pavement skid resistance estimation for autonomous driving. Feature data is collected through the automated vehicle dynamics calibration process, and the selection of features is essentially based on the use of the vehicle dynamics model. To extract the features of one-dimensional timing signals, a temporal one-dimensional convolutional neural network is proposed to improve the stability and robustness of the estimation and minimise the effect of small perturbations and noises. In addition, given the insufficient data that may be faced in practical situations, a transfer learning approach is further proposed to improve the model generalisation. Training data is collected in an autonomous driving simulation platform and the effectiveness of the method is then verified based on this platform. Satisfactory performance is demonstrated for different driving scenarios and under varied road conditions.

Crop Classification with Attention Based BI-LSTM and Temporal Convolution Neural Network Combination for Remote Sensing Breizhcrop Time Series Data

In the modern era, remote sensing data has become increasingly useful for determining land use and coverage requirements. Remote sensing data can be used for a variety of purposes, including the classification of crops. It is possible to aggregate remote sensing data for a specific area over time in order to obtain a more complete picture based on the time series of this data. One example of these types of data is the Breizhcrop dataset, which was collected using satellite images acquired by Sentinel 2 over a period of time. This study aims to investigate a neural network based on attention mechanisms using the BI-LSTM layer in conjunction with Temporal-CNN for the classification of crops. The aim of the research is to find a model for corps classification in image-based time series. In line with this goal, in addition to finding features over time, the presented model also needs to produce high-accuracy features at each time step to increase classification. Utilizing the designed neural network, we seek to find local features with the attention mechanism and general features with a second layer. This neural network was validated on the BreizhCrop dataset and we conclude that it performs better than alternative approaches. The proposed method has been compared with Temporal CNN, Star RNN, and Vanilla LSTM networks and it has obtained better results than the mentioned neural networks. Taking advantage of these local and global features that extract with developed model obtained a high accuracy rate of 82%.

Multi scale encoder-decoder network with Time Frequency Attention and S-TCN for single channel speech enhancement

The goal of speech enhancement is to restore clean speech in noisy environments. Acoustic scenarios with low signal-to-noise ratios (SNR) make it quite challenging to extract the target speech from its noise. In the current study, to enhance noisy speech, we propose a feature recalibration based multi-scale convolutional encoder-decoder architecture with squeeze temporal convolutional networks (S-TCN) bottleneck. Each multi-scale convolutional layer in encoder and decoder is followed by time-frequency attention module (TFA). The recalibration based multi-scale 2D convolution layers are used to extract local and contextual information. Additionally, the recalibration network is equipped with a gating mechanism to control the flow of information among the layers, enabling weighting of the scaled features for noise suppression and speech retention. The fully connected layer (FC) in the bottleneck part of encoder-decoder contains a few neurons, which capture the global information from the multi-scale 2D convolution layer and reduce parameters. A S-TCN, inspired by the popular temporal convolutional neural network (TCNN), is inserted between the encoder and the decoder to model long-term dependencies in speech. The TFA is a highly efficient network component, that operates through two simultaneous attentions, one focused on time frames, and the other on frequency channels. These attentions work together to explicitly exploit positional information to create a two-dimensional attention map to effectively capture the significant time-frequency distribution of speech. Utilizing the common voice dataset, our proposed model consistently enhances results compared to the current benchmarks, as demonstrated by two extensively utilized objective measures PESQ and STOI. The proposed model shows significant improvements, with average PESQ and STOI scores increasing by 45.7% and 23.8% respectively for seen background noises, and by 43.5% and 21.4% for unseen background noises, when compared to the quality of noisy speech. Tests validate that the proposed approach outperforms numerous cutting-edge algorithms.

FBN-TCN: Temporal convolutional neural network based on spatial domain fusion brain networks for affective brain–computer interfaces

Background and objective: Electroencephalogram (EEG) emotion recognition is increasingly popular for visualizing and quantifying brain networks, with potential applications in medical cognition and human–computer interaction. However, single brain space is limited, and multi-channel recognition poses challenges with calculation volume and information redundancy. Therefore, this paper focuses on studying the affective brain–computer interface (aBCI) and critical lead acquisition of the fusion brain network. Method: This paper employs two correlation algorithms to build functional and causal networks. A fusion brain network (FBN) was proposed to capture emotional information in the spatial domain. Additionally, a deep network recognition model (FBN-TCN) is introduced to capture the dynamic changes of spatial domain emotional information over time. A critical channel combination with high emotional correlation among multiple channels is obtained by the brain causal networks. All proposed models were performed on the SEED dataset through subject-dependent and subject-independent experiments. Results: Experimental results demonstrate that FBN can cover more emotional information to improve emotional recognition. The critical channel aBCI achieves comparable recognition performance to the full-channel aBCI, with recognition results of 95.51%/3.15% and 96.34%/1.62% in the Total band, respectively. The generalization and effectiveness of the model were verified through ablation experiments, which involved subject-dependent and subject-independent emotion recognition of DEAP, and emotion recognition of SEED on model block transformation. Conclusion and significance: This paper can simplify the electrode cap device and maximizes the utilization of EEG spatial domain emotional information. It is highly significant for enhancing the performance of aBCI in human–computer interaction.

A method for predicting vibration characteristics of track structure based on rail acceleration and deep learning

ABSTRACT The dynamic test on track structure plays a crucial role in evaluating the operational state. Nonetheless, existing testing approaches involve numerous sensors, resulting in high costs and time consumption. This study proposes an economical, high-precision, and swift solution to predict vibration responses and displacement variations in track structures using rail acceleration. Taking the ordinary track structure and the damping track structure as examples, based on the high correlation between the vibration responses of the rail, track slab and tunnel wall, combined with the proposed Bayesian optimized temporal convolutional neural network model (BOA-TCN), and taking the rail acceleration envelope and vibration level as input, the vibration of tunnel wall and the displacement of rail and track slab are estimated. The results demonstrate that the BOA-TCN model effectively solves the limitation of traditional neural network in accurately capturing vibration characteristics in the frequency range of 1 ~ 100 Hz. The absolute error of the total vibration level prediction is significantly reduced, only 0.74 dB. The single point prediction accuracy of track structure displacement is increased by 80%, the absolute error is 50% of the traditional model, and R2 can be increased to more than 90%.

An adaptive model with dual-dimensional attention for remaining useful life prediction of aero-engine

As an indispensable part in prognostics and health management (PHM) of mechanical systems, the remaining useful life (RUL) estimation has been studied widely and intensively. Recently, with the boom in deep learning, the traditional data-driven prognostic methods like the temporal convolutional neural networks (TCNs), have made remarkable progress in the RUL estimation. However, the conventional TCNs methods do not have a well-established mechanism to parallel weigh the dual-dimensional features of multi-sensory data. Moreover, the conventional TCNs have a fixed network structure which is not flexible to learn deep temporal representations. Here, we propose an adaptive model with dual-dimensional attention for aero-engine RUL prognostics. Specifically, the dual-dimensional multi-head attention (DMHA) module is constructed to assign weights to features in the spatial and temporal dimensions, and then preforms the feature fusion. The mathematical derivations of the DMHA module parameter iterations are performed, which enhances the interpretability of the model. Next, the adaptive TCN (ATCN) is designed to further learn the time correlations of the fused features. Compared with the traditional TCN, the ATCN can adjust the network structure adaptively and extract the deep temporal representations better. Last, the prediction ability of the DMHA-ATCN is verified by conducting experiments on the C-MAPSS dataset.

Data-driven excavation trajectory planning for unmanned mining excavator

In autonomous mining scenarios, excavation trajectory planning plays a significant role since it considerably influences the working performance of the unmanned mining excavator (UME). Aiming at the limited dynamical characterization of traditional theoretical methods that yield unsatisfactory performance in trajectory planning, herein we propose a data-driven excavation trajectory planning framework based on deep learning for improving the operation performance in autonomous excavation scenarios. First, the actual sensing data is collected, and a temporal convolutional recurrent neural network (TCRNN) combining stacked dilated convolutions with an attention-based sequence to sequence (Seq2Seq) module is proposed to predict the digging force accurately. Then, the surrounding material profile is perceived based on Lidar, and a polynomial response surface is performed to point cloud reconstruction. Considering mining efficiency, energy consumption and stability in the excavation process, a multi-objective function is constructed, and a data-driven excavation trajectory planning framework based on TCRNN is established. To obtain the accurate solution of the planning framework, a discretization strategy based on Radau pseudospectral method is developed to model solving to ensure the working performance in autonomous mining. An actual unmanned prototype experiment is performed to investigate the performance of the proposed framework. The results demonstrate the effectiveness and superiority of the proposed framework.

Phase compensation of a continuous-variable quantum key distribution via temporal convolutional neural network

Abstract Although the continuous-variable quantum key distribution (CV-QKD) protocol based on a local local oscillator (LLO) can close all the security loopholes from the transmitted local oscillator (TLO), the phase noise caused by the inaccurate phase reference information limits the performance of the protocol. To reduce the residual phase noise, in this work, we propose a phase estimation and compensation method based on the temporal convolutional neural (TCN) model, where a part of phase information obtained by measuring pilot pulses is employed as the training data and input into the TCN module. With a trained TCN module, the subsequent phase drifts can be more accurately estimated, allowing for better phase compensation and lower phase noise. Numerical analysis shows that the proposed scheme can improve the transmission distance and the secret key rate of the LLO protocol.

High-Order Temporal Convolutional Network for Improving Classification Performance of SSVEP-EEG

Background and objectiveSteady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) aim to detect target frequencies corresponding to specific commands in electroencephalographic (EEG) signals by classification algorithms to achieve the desired control. However, SSVEP signals suffer from low signal-to-noise ratio and large differences in brain activity. Moreover, the existing CNN models have small receptive fields, which make it difficult to receive large range of feature information and limit the effectiveness of classification algorithms. MethodsTo this end, we proposed a high-order temporal convolutional neural network (HOT-CNN) model for enhancing the performance of SSVEP target recognition. Specifically, the SSVEP-EEG signals was divided into equal-length time segments and a time-slice attention module was designed to capture the correlation between time slices. The module improves the local characterization of signals and reduces biological noise interference by automatically assigning high weights to locally relevant temporal sampling cues and lower weights to other temporal cues. Moreover, for global features, a temporal convolutional network module was designed to increases the receptive field of the network and to extract more comprehensive time domain features by using dilated causal convolution. Finally, the fusion and analysis of local and global features are achieved by designing a feature fusion and classification module to accomplish accurate classification of SSVEP signals. ResultsOur method was evaluated on large publicly available datasets containing 35 subjects and 40 categories. Experimental results indicated that HOT-CNN achieved encouraging performance compared with other advanced methods: the highest information transfer rate of 241.01bits/min was obtained using 0.5s stimuli, and the highest average accuracy of 96.39% was obtained using 1.0s stimuli. ConclusionsThe method effectively reinforced the global and local time-domain information and improved the classification performance of SSVEP, which has wide application prospects.

Atmospheric SO2 pollutant prediction using mutual information based TCNN-GRU model for flue gas desulfurization process

Over the past several years, sulfur dioxide (SO2) has raised growing concern in China owing to its adverse impact on atmosphere and human respiratory system. The major contributor to SO2 emissions is flue gas generated by fossil-fired electricity-generating plants, and as a consequence diverse flue gas desulphurization (FGD) techniques are installed to abate SO2 emissions. However, the FGD is a dynamic process with serious nonlinearity and large time delay, making the FGD process modeling problem a formidable one. In our research study, a novel hybrid deep learning model with temporal convolution neural network (TCNN), gated recurrent unit (GRU) and mutual information (MI) technique is proposed to predict SO2 emissions in an FGD process. Among those technique, MI is applied to select variables that are best suited for SO2 emission prediction, while TCNN and GRU are innovatively integrated to capture dynamics of SO2 emission in the FGD process. A real FGD system in a power plant with a coal-fired unit of 1000 MW is used as a study case for SO2 emission prediction. Experimental results show that the proposed approach offers satisfactory performance in predicting SO2 emissions for the FGD process, and outperforms other contrastive predictive methods in terms of different performance indicators.

Hip contact forces can be predicted with a neural network using only synthesised key points and electromyography in people with hip osteoarthritis

ObjectiveTo develop and validate a neural network to estimate hip contact forces (HCF), and lower body kinematics and kinetics during walking in individuals with hip osteoarthritis (OA) using synthesised anatomical key points and electromyography. To assess the capability of the neural network to detect directional changes in HCF resulting from prescribed gait modifications. DesignA calibrated electromyography-informed neuromusculoskeletal model was used to compute lower body joint angles, moments, and HCF for 17 participants with mild-to-moderate hip OA. Anatomical key points (e.g., joint centres) were synthesised from marker trajectories and augmented with bias and noise expected from computer vision-based pose estimation systems. Temporal convolutional and long short-term memory neural networks were trained using leave-one-subject-out validation to predict neuromusculoskeletal modelling outputs from the synthesised key points and measured electromyography data from 5 hip-spanning muscles. ResultsHCF was predicted with an average error of 13.4±7.1% of peak force. Joint angles and moments were predicted with an average root-mean-square-error of 5.3 degrees and 0.10 Nm/kg, respectively. The NN could detect changes in peak HCF that occur due to gait modifications with good agreement with neuromusculoskeletal modelling (r2 = 0.72) and a minimum detectable change of 9.5%. ConclusionThe developed neural network predicted HCF and lower body joint angles and moments in individuals with hip OA using noisy synthesised key point locations with acceptable errors. Changes in HCF magnitude due to gait modifications were predicted with high accuracy. These findings have important implications for implementation of load-modification based gait retraining interventions for people with hip OA in a natural environment (i.e., home, clinic).

Stacked Multiscale Densely Connected Temporal Convolutional Attention Network for Multi-Objective Speech Enhancement in an Airborne Environment

Airborne speech enhancement is always a major challenge for the security of airborne systems. Recently, multi-objective learning technology has become one of the mainstream methods of monaural speech enhancement. In this paper, we propose a novel multi-objective method for airborne speech enhancement, called the stacked multiscale densely connected temporal convolutional attention network (SMDTANet). More specifically, the core of SMDTANet includes three parts, namely a stacked multiscale feature extractor, a triple-attention-based temporal convolutional neural network (TA-TCNN), and a densely connected prediction module. The stacked multiscale feature extractor is leveraged to capture comprehensive feature information from noisy log-power spectra (LPS) inputs. Then, the TA-TCNN adopts a combination of these multiscale features and noisy amplitude modulation spectrogram (AMS) features as inputs to improve its powerful temporal modeling capability. In TA-TCNN, we integrate the advantages of channel attention, spatial attention, and T-F attention to design a novel triple-attention module, which can guide the network to suppress irrelevant information and emphasize informative features of different views. The densely connected prediction module is used to reliably control the flow of the information to provide an accurate estimation of clean LPS and the ideal ratio mask (IRM). Moreover, a new joint-weighted (JW) loss function is constructed to further improve the performance without adding to the model complexity. Extensive experiments on real-world airborne conditions show that our SMDTANet can obtain an on-par or better performance compared to other reference methods in terms of all the objective metrics of speech quality and intelligibility.

Predicting short-term PM2.5 concentrations at fine temporal resolutions using a multi-branch temporal graph convolutional neural network

Predicting PM2.5 concentrations at an hourly temporal resolution in urban areas can provide key information for public health protection. The spatiotemporal dependency among monitoring stations and the spatiotemporal correlations between PM2.5 and relevant factors (e.g. meteorology and emissions) are both essential for such predictions. This study proposes a multi-branch temporal graph convolutional neural network (MB-TGCN) for short-term predictions of PM2.5 concentrations at city monitoring stations. Composed of a set of graph convolutional networks (GCNs) for spatial dependency modeling, a set of gated recurrent units (GRUs) for temporal dependency modeling, and a multi-branch structure for integrating PM2.5 and relevant factors, MB-TGCN aims to accurately predict PM2.5 concentrations by capturing both spatial and temporal relationships through a graph modeling approach. Experiments with an air quality dataset from 35 stations in Beijing showed that MB-TGCN achieved higher accuracy than several deep learning models for various prediction durations ranging from 1 to 12 h. The method described in this study can help enhance the prediction capability of PM2.5 and provide decision support for environment-aware activity planning.

Sequential Learning Network With Residual Blocks: Incorporating Temporal Convolutional Information Into Recurrent Neural Networks

Sequential Learning Network With Residual Blocks: Incorporating Temporal Convolutional Information Into Recurrent Neural Networks

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

AbstractExposure to varied noisy environments impairs the recognition performance of artificial intelligence‐based speech recognition technologies. Degraded‐performance services can be utilized as limited systems that assure good performance in certain environments, but impair the general quality of speech recognition services. This study introduces an audiovisual speech recognition (AVSR) model robust to various noise settings, mimicking human dialogue recognition elements. The model converts word embeddings and log‐Mel spectrograms into feature vectors for audio recognition. A dense spatial–temporal convolutional neural network model extracts features from log‐Mel spectrograms, transformed for visual‐based recognition. This approach exhibits improved aural and visual recognition capabilities. We assess the signal‐to‐noise ratio in nine synthesized noise environments, with the proposed model exhibiting lower average error rates. The error rate for the AVSR model using a three‐feature multi‐fusion method is 1.711%, compared to the general 3.939% rate. This model is applicable in noise‐affected environments owing to its enhanced stability and recognition rate.

Embedded Spatial–Temporal Convolutional Neural Network Based on Scattered Light Signals for Fire and Interferential Aerosol Classification

Photoelectric smoke detectors are the most cost-effective devices for very early warning fire alarms. However, due to the different light intensity response values of different kinds of fire smoke and interference from interferential aerosols, they have a high false-alarm rate, which limits their popularity in Chinese homes. To address these issues, an embedded spatial–temporal convolutional neural network (EST-CNN) model is proposed for real fire smoke identification and aerosol (fire smoke and interferential aerosols) classification. The EST-CNN consists of three modules, including information fusion, scattering feature extraction, and aerosol classification. Moreover, a two-dimensional spatial–temporal scattering (2D-TS) matrix is designed to fuse the scattered light intensities in different channels and adjacent time slices, which is the output of the information fusion module and the input for the scattering feature extraction module. The EST-CNN is trained and tested with experimental data measured on an established fire test platform using the developed dual-wavelength dual-angle photoelectric smoke detector. The optimal network parameters were selected through extensive experiments, resulting in an average classification accuracy of 98.96% for different aerosols, with only 67 kB network parameters. The experimental results demonstrate the feasibility of installing the designed EST-CNN model directly in existing commercial photoelectric smoke detectors to realize aerosol classification.

Multistage spatio-temporal attention network based on NODE for short-term PV power forecasting

Photovoltaic (PV) power has attracted widespread attention from many countries around the world due to its clean and renewable characteristics. To ensure the stable operation of the power system, accurate PV power forecasting has become a mandatory and challenging task. Currently, deep learning methods have become a vital approach in the field of PV power forecasting. In this work, a multistage attention neural network based on neural ordinary differential equation (MANODE) is proposed to address the main limitations of previous deep learning methods applied to PV power forecasting. Based on the neural ordinary differential equation (NODE), MANODE optimizes the long short-term memory network (LSTM) and temporal convolutional neural network (TCN), and combines the attention mechanism to achieve fine-grained spatio-temporal information extraction of PV series. In addition, the proposed MANODE model is applied to three different PV series collected from the Alice Springs meteorological station. Compared to previous state-of-the-art methods, the proposed method reduces the PV power forecasting error by at least 12.05%, 13.15%, and 9.71% on three different PV datasets, in terms of mean absolute error metric. The average errors of the MANODE method in four-hour-ahead PV power forecasting on the three datasets are 0.321, 0.350, and 0.567.

Micro aneurysm detection using optimized residual-based temporal attention Convolutional Neural Network with Inception-V3 transfer learning.

In this manuscript micro aneurysm detection using residual-based temporal attention Convolutional Neural Network (CNN) with Inception-V3 transfer learning optimized with equilibrium optimization algorithm (MA-RTCNN-Inception V3-EOA) is proposed. The proposed research work contains four phases: (1) pre-processing, (2) segmentation, (3) post-processing, and (4) classification. At first, guided box filtering for contrast enhancement and background exclusion of input image. The proposed MA-RTCNN-Inception V3-EOA based classification framework is implemented in MATLAB using several performances evaluating metrics like precision, sensitivity, f-measure, specificity, accuracy, classification error rate, and Matthews's correlation coefficient and RoC analysis. The experimental outcome demonstrates that the proposed method provides 23.56%, 14.99%, and 21.37% higher accuracy and 31.26%, 57.69%, and 21.14% minimum classification error rate compared to existing methods, such as diabetic retinopathy identification utilizing prognosis of micro aneurysm and early diagnosis for non-proliferative diabetic retinopathy depending on deep learning approaches (DRD-CNN-NPDR), a magnified adaptive feature pyramid network for automatic micro aneurysms identification (MAFPN-AMD-MAFP-Net) respectively. RESEARCH HIGHLIGHTS: Micro aneurysm detection using residual-based temporal attention Convolutional Neural Network (CNN) is proposed. To get rid of the retina background, guided box filtering is applied. COAT is used for segmenting the images into smaller parts RTCNN is used for accurate micro aneurysms disease classification. RT-CNN algorithm successfully identifies the micro aneurysms using EOA.

Impact force localization and reconstruction via gated temporal convolutional network

Techniques for identifying impact forces are crucial for improving the operational reliability and safety of aerospace composite structures. To address the challenges of theoretical modeling that arise from the complexity of composite structures, we propose a Gated Temporal Convolutional Network (GTCN) method. This method establishes inverse mapping relationships between structural vibration responses and impact forces. The GTCN increases the receptive field through dilated convolution, concurrently employing gating mechanisms to extract pivotal features from input signals in the vertical direction. Simulations and experiments are conducted to robustly validate the efficacy and applicability of the proposed impact force identification method. The simulation results demonstrate that the GTCN method is capable of obtaining robust impact force localization and reconstruction results, even amidst conditions of elevated noise levels. The average localization error of the GTCN is kept within 20 mm even at a noise level of 5 dB. Meanwhile, the GTCN method achieves satisfactory identification results for different types of impact forces (Gaussian, semi-cosine, and sawtooth), with an average localization accuracy above 95%. The applicability of the GTCN method to real-world structure is verified in the experiments. The average relative error of the reconstructed forces by GTCN is less than 15%. GTCN achieves stable identification of impact forces when the number of sensors is larger than 2. In both simulations and experiments, the GTCN performs better than Temporal Convolutional Network (TCN) and Convolutional Neural Network (CNN) with higher identification accuracy. The proposed method demonstrates considerable potential for application in the health monitoring of aircraft composite structures.