Residual LSTM Research Articles

Achieving More with Less: A Lightweight Deep Learning Solution for Advanced Human Activity Recognition (HAR).

Human activity recognition (HAR) is a crucial task in various applications, including healthcare, fitness, and the military. Deep learning models have revolutionized HAR, however, their computational complexity, particularly those involving BiLSTMs, poses significant challenges for deployment on resource-constrained devices like smartphones. While BiLSTMs effectively capture long-term dependencies by processing inputs bidirectionally, their high parameter count and computational demands hinder practical applications in real-time HAR. This study investigates the approximation of the computationally intensive BiLSTM component in a HAR model by using a combination of alternative model components and data flipping augmentation. The proposed modifications to an existing hybrid model architecture replace the BiLSTM with standard and residual LSTM, along with convolutional networks, supplemented by data flipping augmentation to replicate the context awareness typically provided by BiLSTM networks. The results demonstrate that the residual LSTM (ResLSTM) model achieves superior performance while maintaining a lower computational complexity compared to the traditional BiLSTM model. Specifically, on the UCI-HAR dataset, the ResLSTM model attains an accuracy of 96.34% with 576,702 parameters, outperforming the BiLSTM model's accuracy of 95.22% with 849,534 parameters. On the WISDM dataset, the ResLSTM achieves an accuracy of 97.20% with 192,238 parameters, compared to the BiLSTM's 97.23% accuracy with 283,182 parameters, demonstrating a more efficient architecture with minimal performance trade-off. For the KU-HAR dataset, the ResLSTM model achieves an accuracy of 97.05% with 386,038 parameters, showing comparable performance to the BiLSTM model's 98.63% accuracy with 569,462 parameters, but with significantly fewer parameters.

DRLSTM-DA: a novel aeroengine remaining useful life prediction method based on deep optimization of multiple sensors information

Abstract One of the crucial tasks in Remaining Useful Life (RUL) prediction is to effectively extract key sensor information from numerous sensor signals. In this article, a novel dual-attention enhanced deep residual LSTM (DRLSTM-DA) is developed to deeply optimize multidimensional sensor signals. First, two LSTM layers are designed to compress and reconstruct representative degradation information from input multidimensional time series data, to generate a new feature space. Second, a novel channel adaptive soft threshold module is designed to assign different weights according to the importance of different sensor information, and simultaneously eliminate the noise information in the signal. Thirdly, a temporal attention mechanism is designed to automatically highlight moments containing important decay information while suppressing unimportant moments. Finally, compared with other SOTA methods, our architecture achieves RMSE of 11.55, 13.74, 11.25, and 14.19 on four sub-datasets of the C-MAPSS dataset, with scores of 234.24, 465.49, 202.23, and 537.66, respectively. Meanwhile, in real aeroengine operation dataset, our architecture achieved the smallest RMSE (8.62). These results all validate the good predictive performance of our model.

Residual and bidirectional LSTM for epileptic seizure detection.

Electroencephalogram (EEG) plays a pivotal role in the detection and analysis of epileptic seizures, which affects over 70 million people in the world. Nonetheless, the visual interpretation of EEG signals for epilepsy detection is laborious and time-consuming. To tackle this open challenge, we introduce a straightforward yet efficient hybrid deep learning approach, named ResBiLSTM, for detecting epileptic seizures using EEG signals. Firstly, a one-dimensional residual neural network (ResNet) is tailored to adeptly extract the local spatial features of EEG signals. Subsequently, the acquired features are input into a bidirectional long short-term memory (BiLSTM) layer to model temporal dependencies. These output features are further processed through two fully connected layers to achieve the final epileptic seizure detection. The performance of ResBiLSTM is assessed on the epileptic seizure datasets provided by the University of Bonn and Temple University Hospital (TUH). The ResBiLSTM model achieves epileptic seizure detection accuracy rates of 98.88-100% in binary and ternary classifications on the Bonn dataset. Experimental outcomes for seizure recognition across seven epilepsy seizure types on the TUH seizure corpus (TUSZ) dataset indicate that the ResBiLSTM model attains a classification accuracy of 95.03% and a weighted F1 score of 95.03% with 10-fold cross-validation. These findings illustrate that ResBiLSTM outperforms several recent deep learning state-of-the-art approaches.

Predicting tidal level in tropical Eastern Bintan waters using residual long short-term memory

<span lang="EN-US">The sea brings many benefits for society, especially for a maritime country such as Indonesia. The potential in various sectors is limited only by the willingness of a party to invest in it. One such investment is in learning the knowledge and information that can be gathered from the sea, and even predicting its behavior with enough data. Using a residual LSTM algorithm, we will predict the tidal level in eastern Bintan island, a tropical island on the tip of Malay peninsula. The dataset is acquired from two sensor points in eastern Bintan coast from July 2018 to June 2019 for a span of one year, giving a total of 7,961 data points. The residual LSTM model consists of a residual wrapper with two consecutive LSTM layers and one dense layer. The model is also compared with variations of LSTM and RNN models. The result of the residual LSTM model has an MAE value of 0.1495 cm and an RMSE value of 0.3353 cm, compared to the baseline model’s 1.1148 cm and 1.4107 cm respectively. The model also has an RMSE value improvement of 76.23% compared to the base model.</span>

Velocity-aware spatial-temporal attention LSTM model for inverse dynamic model learning of manipulators.

An accurate inverse dynamics model of manipulators can be effectively learned using neural networks. However, further research is required to investigate the impact of spatiotemporal variations in manipulator motion sequences on network learning. In this work, the Velocity Aware Spatial-Temporal Attention Residual LSTM neural network (VA-STA-ResLSTM) is proposed to learn a more accurate inverse dynamics model, which uses a velocity-aware spatial-temporal attention mechanism to extract dynamic spatiotemporal features selectively from the motion sequence of the serial manipulator. The multi-layer perception (MLP) attention mechanism is adopted to capture the correlation between joint position and velocity in the motion sequence, and the state correlation between hidden units in the LSTM network to reduce the weight of invalid features. A velocity-aware state fusion approach of LSTM network hidden units' states is proposed, which utilizes variation in joint velocity to adapt to the temporal characteristics of the manipulator dynamic motion, improving the generalization and accuracy of the neural network. Comparative experiments have been conducted on two open datasets and a self-built dataset. Specifically, the proposed method achieved an average accuracy improvement of 61.88% and 43.93% on the two different open datasets and 71.13% on the self-built dataset compared to the LSTM network. These results demonstrate a significant advancement in accuracy for the proposed method. Compared with the state-of-the-art inverse dynamics model learning methods of manipulators, the modeling accuracy of the proposed method in this paper is higher by an average of 10%. Finally, by visualizing attention weights to explain the training procedure, it was found that dynamic modeling only relies on partial features, which is meaningful for future optimization of inverse dynamic model learning methods.

Positional multi-length and mutual-attention network for epileptic seizure classification

The automatic classification of epilepsy electroencephalogram (EEG) signals plays a crucial role in diagnosing neurological diseases. Although promising results have been achieved by deep learning methods in this task, capturing the minute abnormal characteristics, contextual information, and long dependencies of EEG signals remains a challenge. To address this challenge, a positional multi-length and mutual-attention (PMM) network is proposed for the automatic classification of epilepsy EEG signals. The PMM network incorporates a positional feature encoding process that extracts minute abnormal characteristics from the EEG signal and utilizes a multi-length feature learning process with a hierarchy residual dilated LSTM (RDLSTM) to capture long contextual dependencies. Furthermore, a mutual-attention feature reinforcement process is employed to learn the global and relative feature dependencies and enhance the discriminative abilities of the network. To validate the effectiveness PMM network, we conduct extensive experiments on the public dataset and the experimental results demonstrate the superior performance of the PMM network compared to state-of-the-art methods.

Lokatt: a hybrid DNA nanopore basecaller with an explicit duration hidden Markov model and a residual LSTM network

BackgroundBasecalling long DNA sequences is a crucial step in nanopore-based DNA sequencing protocols. In recent years, the CTC-RNN model has become the leading basecalling model, supplanting preceding hidden Markov models (HMMs) that relied on pre-segmenting ion current measurements. However, the CTC-RNN model operates independently of prior biological and physical insights.ResultsWe present a novel basecaller named Lokatt: explicit duration Markov model and residual-LSTM network. It leverages an explicit duration HMM (EDHMM) designed to model the nanopore sequencing processes. Trained on a newly generated library with methylation-free Ecoli samples and MinION R9.4.1 chemistry, the Lokatt basecaller achieves basecalling performances with a median single read identity score of 0.930, a genome coverage ratio of 99.750%, on par with existing state-of-the-art structure when trained on the same datasets.ConclusionOur research underlines the potential of incorporating prior knowledge into the basecalling processes, particularly through integrating HMMs and recurrent neural networks. The Lokatt basecaller showcases the efficacy of a hybrid approach, emphasizing its capacity to achieve high-quality basecalling performance while accommodating the nuances of nanopore sequencing. These outcomes pave the way for advanced basecalling methodologies, with potential implications for enhancing the accuracy and efficiency of nanopore-based DNA sequencing protocols.

Residual LSTM neural network for time dependent consecutive pitch string recognition from spectrograms: a study on Turkish classical music makams

Residual LSTM neural network for time dependent consecutive pitch string recognition from spectrograms: a study on Turkish classical music makams

Prediction of influent wastewater quality based on wavelet transform and residual LSTM

Accurate prediction on influent wastewater quality is of great importance to energy saving and chemical dosage reduction of wastewater treatment plants (WWTPs). However, the existing methods ignore the data noise caused by water sensors working in harsh conditions and the intrinsic variable dynamics inherent in the time series of wastewater quality. To tackle this problem, we propose a novel approach called wt-ResLSTM (wavelet transform and Residual Long Short-Term Memory) to predict the influent wastewater quality. Specifically, we adopt wavelet transform and semi-soft thresholding to remove the noise from influent wastewater quality data adaptively. Then, we use autoencoder to learn the latent representation of the recent fluctuation of wastewater quality to capture its transient uncertainty. Further, the residual LSTM is adopted to learn both the long-term and short-term sequential dependencies of influent wastewater quality from the historical wastewater quality and the latent representation of its recent fluctuation. Experiments on the dataset from a large-scale urban WWTP in Beijing demonstrate that the proposed wt-ResLSTM approach outperforms state-of-the-art techniques in predicting influent wastewater quality in terms of level accuracy and directional accuracy.

ICaps-ResLSTM: Improved capsule network and residual LSTM for EEG emotion recognition

Electroencephalography (EEG) emotion recognition is an important task for brain–computer interfaces. The time, frequency, and spatial domains of EEG signals have been widely studied. However, these methods often ignore the spatial and temporal correlations in dual modules, resulting in insufficient emotional representations. In this paper, a dual module EEG emotion recognition method based on an improved capsule network and residual Long-Short Term Memory (ResLSTM) is proposed. Using an improved capsule network as the spatial module is more advantageous in learning specific EEG spatial representations. The ResLSTM of the temporal module inherits the information flow from the upper spatial module and conducts complementary learning of the spatiotemporal dual module features through residual connections, thus obtaining more discriminative EEG features and ultimately boosting the classification capabilities of the model. The average accuracy of arousal, valence, and dominance on the DEAP dataset reached 98.06%, 97.94%, and 98.15%, respectively. The DREAMER dataset’s average accuracy of arousal, valence, and dominance reached 94.97%, 94.71%, and 94.96%, respectively. The results of our experiments indicate that our method outperforms state-of-the-art approaches.

Deep Domain-Invariant Long Short-Term Memory Network for Partial Discharge Localization in Gas-Insulated Switchgear

Localization based on differences in the timing of an ultrahigh-frequency signal is difficult to apply in actual settings because of the strict requirement for accurate calculation of the difference in arrival time between discharge pulses. With the development of digital twinning, the operational state of equipment is mapped onto a digital virtual model, and large-scale simulation data can be obtained to assist in localization of partial discharge (PD) in gas-insulated switchgear (GIS). With this aim, this paper proposes a novel deep domain-invariant long short-term memory network (LSTM) for GIS PD localization. Firstly, an attention residual LSTM is constructed to automatically extract discriminative features and effectively obtain the temporal and spatial dependences between features. Collaborative domain adaptation is proposed to transfer the diagnostic knowledge learned from large-scale simulation data to GIS PD localization. Via maximum mean discrepancy-based domain adaptation and domain adversarial training, discrepancies between samples in the two domains are reduced so as to achieve a more perfect match. The domain-invariant LSTM achieved an average localization accuracy of less than 11 cm, and the standard deviation of the error reached 8.15. The domain-invariant LSTM thus provides a novel solution for the high-precision and robust localization of PD in GIS.

GRLSTM: Trajectory Similarity Computation with Graph-Based Residual LSTM

The computation of trajectory similarity is a crucial task in many spatial data analysis applications. However, existing methods have been designed primarily for trajectories in Euclidean space, which overlooks the fact that real-world trajectories are often generated on road networks. This paper addresses this gap by proposing a novel framework, called GRLSTM (Graph-based Residual LSTM). To jointly capture the properties of trajectories and road networks, the proposed framework incorporates knowledge graph embedding (KGE), graph neural network (GNN), and the residual network into the multi-layer LSTM (Residual-LSTM). Specifically, the framework constructs a point knowledge graph to study the multi-relation of points, as points may belong to both the trajectory and the road network. KGE is introduced to learn point embeddings and relation embeddings to build the point fusion graph, while GNN is used to capture the topology structure information of the point fusion graph. Finally, Residual-LSTM is used to learn the trajectory embeddings.To further enhance the accuracy and robustness of the final trajectory embeddings, we introduce two new neighbor-based point loss functions, namely, graph-based point loss function and trajectory-based point loss function. The GRLSTM is evaluated using two real-world trajectory datasets, and the experimental results demonstrate that GRLSTM outperforms all the state-of-the-art methods significantly.

Dynamic Prediction of Longitudinal Settlement of Existing Tunnel Using ConvRes-DLinear Model with Integration of Undercrossing Construction Process Information

Undercrossing construction can cause severe structural deformation of the above existing tunnel in operation. The induced longitudinal differential settlement between the segments can pose a huge risk to running subways, hence it is of great importance to monitor and predict the settlement. Within this study, a Wireless Sensor Network (WSN) system was implemented to obtain hourly monitoring data of settlement from the very beginning of undercrossing to post construction period. An improved direct multi-step (DMS) forecasting model called ConvRes-DLinear is proposed, which fuses monitoring data with time and process encoding bias to deeply extract and learn temporal correlation of time series. A residual LSTM model is also constructed to compare the accuracy of the improved DLinear model. The training and testing experiment on the monitoring data of longitudinal settlement obtained by WSN system shows that the ConvRes-DLinear model with time and process encoding bias performs surprisingly well with a minimum prediction error. The features of the proposed model are discussed to make the results explainable. The monitoring system and time series forecasting model proposed in this study have a guiding significance for the monitoring and prediction of longitudinal differential settlement of tunnels under environmental disturbance.

Condition Monitoring and Fault Detection of Wind Turbine Driveline With the Implementation of Deep Residual Long Short-Term Memory Network

In view of the wide application of wind power, it is critically essential to develop solutions to the high fault rate and the long mean time to repair (MTTR). Multi-sensor fusion methods have been widely applied in condition monitoring and anomaly detection of industrial installation. Aimed at the wind turbine driveline, a new method utilizing the deep residual LSTM network with Attention model (ResLSTM-AM) is proposed in this paper. Firstly, data from multi-sensor fusion systems like supervisory control and data acquisition (SCADA) is preprocessed by quartile data cleaning and corresponding selection using the calculation of maximal information coefficient (MIC) to enhance its reliability. Secondly, we establish the neural network by the deep residual network (ResNet), long short-term memory network (LSTM), and attention mechanism (AM) for time series forecasting of wind turbines. The newly added residual connecting architecture in LSTM layers ensures a higher learning rate in feature extraction and provides a more flexible data flow both in forward and backward signals. Additionally, the attention mechanism after a LSTM layer amplifies the influence of vital data. The model gets validated based on historical SCADA data from actual wind turbines containing healthy conditions and anomaly conditions, showing ResNet contributes more. Based on the root-mean-square error (RMSE), the alarming threshold is calculated by an exponential weighted moving average (EWMA). For two chosen variables, the proposed method is confirmed to be efficient and reliable with the accuracy of 0.9945 and 0.9880 in the results of trials and comparisons with other existing models.

Role of Machine Learning Algorithms for Wind Power Generation Prediction in Renewable Energy Management

The electrical energy demand is growing every day. Fossil fuel-based electrical power generation pollutes the environment. So, to fulfil the electrical energy demand, clean renewable energy sources like solar and wind are emerging as viable alternatives. But the amount of power generated from solar and wind energy sources is varying in magnitude and is intermittent in nature. So, predicting solar and wind power generation is indispensable for efficient planning and controlling of a renewable energy system. This paper aims to identify a machine learning model suitable for time-series prediction of wind power and also surveys the predominantly existing machine learning followed by the deep learning approaches. The main objective of this study is to solve a real-life problem in the renewable energy sector by accurately estimating the amount of power generation production per hour by applying machine learning techniques using historical wind power energy production data. Hence, historical wind data set with input parameters like wind speed, wind direction air temperature, pressure, and output parameters of wind power generation is considered for training and testing using mean absolute error for evaluating the prediction accuracy. Layers from simple to complex deep layers are trained and tested for prediction accuracy, i.e. baseline, linear, dense, multidense, convolution neural networks, and long short-term memory. Investigating further, LSTM and Residual LSTM prove to have higher mean absolute prediction accuracy of 0.0987 and 0.0958. This eventually enlightens the wind power forecasting models and new paths for further research applications.

Res-LGAN for human motion style transfer with more features preserved

Human motion style transfer is a technique that aims to apply a desired style to neutral motions, which is an essential aspect of motion generation and retargeting. With the advancement of deep learning networks, significant progress has been made in this field. However, one of the main challenges is preserving the essential features of the original motions, such as velocities and trace, during the style transfer process. To overcome this challenge, we have proposed a novel method called Residual LSTM Generative Adversarial Networks (Res-LGAN) for motion style transfer. The Res-LGAN models consist of a transfer network and a refinement network, which work together to generate smooth and natural stylized motions while preserving key features of the original motions. Additionally, we have introduced a reconstruction loss term to ensure the stylized motions closely retain the features of the original motions. Our experiments demonstrate that the proposed Res-LGAN model outperforms existing state-of-the-art models by generating high-quality stylized motions while preserving the original motion features. To the best of our knowledge, Res-LGAN is the leading method for preserving original content features during motion style transferring.

Residual long short-term memory network with multi-source and multi-frequency information fusion: An application to China's stock market

The most widely used model in stock price forecasting is the long short-term memory network (LSTM). However, LSTM has its limitations, as it does not recognize and extract features well and has a representational bottleneck. Furthermore, the factors affecting stock prices are multi-source and multi-frequency information, making neural network models difficult to handle. In this paper, we introduce a feature fusion residual LSTM (FFRL) model to answer these two questions – how to compensate for the three limitations of LSTM and how to fuse the multi-source and multi-frequency information. FFRL consists of three modules to improve the three limitations of LSTM, namely the feature selection module, feature extraction module, and residual module. To learn features from multi-source and multi-frequency information, FFRL applies the feature selection module to emphasize important features and the feature extraction module to extract deeper features. We demonstrate significant performance improvements of FFRL over comparison models, ablation networks, and visualization methods on a variety of Chinese stocks.

Application of a hybrid model based on the Prophet model, ICEEMDAN and multi-model optimization error correction in metal price prediction

Nonferrous metals are the basic materials and key strategic reserve resources for national economic development. The fluctuation of metal prices seriously affects the development of modern industry and the global economy. Therefore, accurate and stable nonferrous metal price prediction is very important and meaningful. In this paper, a relatively novel prediction system is proposed to predict the price of metals. The hybrid model of the Prophet model, ICEEMDAN and multi-model optimization error correction is applied to predict the prices of the metals zinc, aluminum, copper and gold. The Prophet model can predict the data more concisely by combining the research background and statistical knowledge, but there are also some nonlinear residual sequences in the prediction. Therefore, ICEEMDAN is used to decompose the residual sequence, and ARIMA, ELMAN, BPNN, LSTM and NAR are applied to predict the decomposed subsequences. Then, the optimal prediction results in each subsequence are selected to add the residual prediction value. Finally, the prediction value of the original price is obtained by adding the Prophet model and the residual prediction value. The price series of four metals from the London Metal Exchange and Investing are selected as the research object. Through two groups of experiments, the proposed model is compared with six single models and six hybrid models. At the same time, to further verify the performance and the reliability of the proposed model, DM detection and model confidence set trimming are used. Through the evaluation index and the performance test of the model, the proposed model's superiority and reliability in metal price prediction are effectively proven.

Evaluating Residual LSTM approach for predicting missing sensor data for IoMT


 
 
 Currently we are living in a digital age due to advancements in the fields like Internet of Things, Artificial Intelligence and Big Data. Especially IoT applications like Smart Home, Smart Watch, Smart Farming, Smart Retail and Smart Parking are seen everywhere around us. These applications are termed Smart due to their ability of self-decision making and monitoring surrounding environment. Data is the most precious assets that requires to make IoT application smart. But multiple times sensor data is found missing or noisy. Due to various reasons like sensor malfunction, sensor maintenance and poor internet communication between devices leads to rise of missing values. In majority cases Missing Completely at Random (MCAR) type missing data are found. Hence, we propose a Residual LSTM model approach for more accurate prediction of missing sensor data on TILES dataset consisting features like Breathing Depth, Breathing Rate and Heart Rate. Compared to approaches like Mean, Median, K-Nearest Neighbours’ and Bidirectional Recurrent Neural Network the Residual Long Short-Term Memory model yields better accuracy which is of huge importance in IoMT application.
 
 
 
 
 
 
 
 

IGCRRN: Improved Graph Convolution Res-Recurrent Network for spatio-temporal dependence capturing and traffic flow prediction

Accurate traffic flow prediction is critical for traffic management and route guidance, enabling urban traffic to be free-flowing conditions and maximizing transport efficiency. In current prediction methods, the simple and fixed spatial graph only uses the prior knowledge of the traffic network, resulting in weak prediction performance. This paper proposes an Improved Graph Convolution Res-Recurrent Network (IGCRRN), which relies on uncertain spatio-temporal information for traffic flow prediction. In particular, a spatial dependence matrix that combines the origin graph matrix and the data-generated embedding node matrix is created. In this way, the spatial connection relationship can be obtained from the static graph information and changing traffic flow series, making the improved graph convolution block infer and quantify the different contributions in both spatial dependence and temporal dependence in a data-driven manner. In addition, the residual structure is employed to model the multi-level spatial dependence, and the IGCRRN-cell units based on the residual connection block and LSTM are designed to make the model automatically capture the spatio-temporal dependence in the traffic flow sequence. Experiments are conducted on two real traffic datasets, and the experiment results show that our proposed spatial dependence matrix can investigate the valuable information and consider the heterogeneity in the traffic flow. The IGCRRN model outperforms the baseline and state-of-the-art methods in prediction performance.