Stacked Recurrent Neural Networks Research Articles

Predicting the closing price of cryptocurrency Ethereum

Given that cryptocurrencies are now involved in nearly every financial transaction due to their widespread acceptance as an alternative method of payment and currency exchange, researchers and economists have increased opportunities to analyze cryptocurrency prices. Over time, predicting the daily closing price of Ethereum has been challenging for investors, traders, and investment banks because of its significant price volatility. The daily closing price of cryptocurrency is crucial for trading or investing in Ethereum. This report aims to conduct a comparative analysis of the predictive performance of deep machine learning algorithms within a stacking ensemble modeling framework, utilizing daily historical price data of Ethereum from Coindesk, tweets from Twitter spanning from August 1, 2022, to August 8, 2022, and five additional covariates (closing price lag1, closing price lag2, noltrend, daytype, and month) derived from Ethereum's closing price. Seven models are employed to forecast the daily closing price of Ethereum: recurrent neural network, ensemble stacked recurrent neural network, gradient boosting machine, generalized linear model, distributed random forest, deep neural networks, and a stacked ensemble of gradient boosting machine, generalized linear model, distributed random forest, and deep neural networks. The primary evaluation metric is the mean absolute error (MAE). Based on MAE, the RNN forecasts outperform the other models in this study, achieving an MAE of 0.0309.

Blockchain with optimal deep learning assisted secure data sharing and classification on future healthcare systems

In the future of healthcare, Blockchain (BC) technology holds immense potential for improving the security and privacy of data. By allowing the secure and immutable storage of medical files and healthcare-related transactions, BC ensured that sensitive medical data remains tamper-proof and open only to authorized parties. Patients have greater control over their data's development, revoking or granting access as required, but healthcare workers can streamline data sharing and ensure the integrity of important data. The decentralized nature of BC networks decreases the risk of centralized data breaches, eventually fostering trust and transparency in the healthcare ecosystems. Conversely, deep learning (DL) has great to revolutionize healthcare diagnostics in the future, offering quick and extremely accurate estimates of medical conditions. This technology has greatly enhanced patient solutions, decreased medical expenses, and improved the burden on medical staff by providing appreciated insights into an extensive range of conditions, from cancer to neurological disorders. With this stimulus, this study presents a novel BC with optimal DL-based secure data sharing and classification (BCODL-SDSC) technique in the future healthcare system. The goal of the BCODL-SDSC technique is to secure and thoroughly examine healthcare data using BC and DL techniques. Primarily, the BCODL-SDSC technique enables BC technology to store and maintain the patient’s data from the procedure of several transactions and enable access control to the various stakeholders. For the security of the medical images, the BCODL-SDSC technique applies the Fractional Order Lorenz system (FOLS) based encryption technique with tuna swarm optimization (TSO) algorithm based optimal key generation process. Finally, a multi-stage process performs the classification of the medical images: MobileNetv1 feature extractor, artificial rabbit’s optimization (ARO) based hyperparameter tuning, and stacked recurrent neural network (SRNN) based classification. The experimental outcome of the BCODL-SDSC technique was examined on a benchmark medical image database. An extensive comparative study reported that the BCODL-SDSC technique reaches an effective performance with other models with a maximum accuracy of 99.11%.

RNNCon: Contribution Coverage Testing for Stacked Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are applied in safety-critical fields such as autonomous driving, aircraft collision detection, and smart credit. They are highly susceptible to input perturbations, but little research on RNN-oriented testing techniques has been conducted, leaving a threat to a large number of sequential application domains. To address these gaps, improve the test adequacy of RNNs, find more defects, and improve the performance of RNNs models and their robustness to input perturbations. We aim to propose a test coverage metric for the underlying structure of RNNs, which is used to guide the generation of test inputs to test RNNs. Although coverage metrics have been proposed for RNNs, such as the hidden state coverage in RNN-Test, they ignore the fact that the underlying structure of RNNs is still a fully connected neural network but with an additional "delayer" that records the network state at the time of data input. We use the contributions, i.e., the combination of the outputs of neurons and the weights they emit, as the minimum computational unit of RNNs to explore the finer-grained logical structure inside the recurrent cells. Compared to existing coverage metrics, our research covers the decision mechanism of RNNs in more detail and is more likely to generate more adversarial samples and discover more flaws in the model. In this paper, we redefine the contribution coverage metric applicable to Stacked LSTMs and Stacked GRUs by considering the joint effect of neurons and weights in the underlying structure of the neural network. We propose a new coverage metric, RNNCon, which can be used to guide the generation of adversarial test inputs. And we design and implement a test framework prototype RNNCon-Test. 2 datasets, 4 LSTM models, and 4 GRU models are used to verify the effectiveness of RNNCon-Test. Compared to the current state-of-the-art study RNN-Test, RNNCon can cover a deeper decision logic of RNNs. RNNCon-Test is not only effective in identifying defects in Deep Learning (DL) systems but also in improving the performance of the model if the adversarial inputs generated by RNNCon-Test are filtered and added to the training set to retrain the model. In the case where the accuracy of the model is already high, RNNCon-Test is still able to improve the accuracy of the model by up to 0.45%.

Estimation of Lithium-ion Battery Discharge Capacity by Integrating Optimized Explainable-AI and Stacked LSTM Model

Accurate lithium-ion battery state of health evaluation is crucial for correctly operating and managing battery-based energy storage systems. Experimental determination is problematic in these applications since standard functioning is necessary. Machine learning techniques enable accurate and effective data-driven predictions in such situations. In the present paper, an optimized explainable artificial intelligence (Ex-AI) model is proposed to predict the discharge capacity of the battery. In the initial stage, three deep learning (DL) models, stacked long short-term memory networks (stacked LSTMs), gated recurrent unit (GRU) networks, and stacked recurrent neural networks (SRNNs) were developed based on the training of six input features. Ex-AI was applied to identify the relevant features and further optimize Ex-AI operating parameters, and the jellyfish metaheuristic optimization technique was considered. The results reveal that discharge capacity was better predicted when the jellyfish-Ex-AI model was applied. A very low RMSE of 0.04, MAE of 0.60, and MAPE of 0.03 were observed with the Stacked-LSTM model, demonstrating our proposed methodology’s utility.

Stacked recurrent neural network for botnet detection in smart homes

Internet of Things (IoT) devices in Smart Home Network (SHN) are highly vulnerable to complex botnet attacks. In this paper, we investigate the effectiveness of Recurrent Neural Network (RNN) to correctly classify network traffic samples in the minority classes of highly imbalanced network traffic data. Multiple layers of RNN are stacked to learn the hierarchical representations of highly imbalanced network traffic data with different levels of abstraction. We evaluate the performance of Stacked RNN (SRNN) model with Bot-IoT dataset. Results show that SRNN outperformed RNN in all classification scenarios. Specifically, SRNN model learned the discriminating features of highly imbalanced network traffic samples in the training set with better representations than RNN model. Also, SRNN model is more robust and it demonstrated better capability to effectively handle over-fitting problem than RNN model. Furthermore, SRNN model achieved better generalization ability in detecting network traffic samples of the minority classes.

Features injected recurrent neural networks for short-term traffic speed prediction

Accurate traffic speed forecasting is critical in advanced transportation management and traveler route planing. Considering the important influences of spatial–temporal factors and excellent performance of recurrent neural networks (RNNs) in the field of time series analyzing, in this paper, the features injected recurrent neural networks (FI-RNNs) were proposed, which combines sequential time data with contextual factors to mine the potential relationship between traffic state and its context. In this model, a stacked RNN was used to learn the sequence features of traffic data. Meanwhile, a sparse Autoencoder was trained to expand the contextual features, which are high-level coding and abstract representations of contextual factors. Then an merging mechanism which injects contextual features into sequence features was explored to generate fusion features. Finally, the new fused features were fed to the predictor to learn the traffic patterns and predict future speed. Case studies based on two real-world data sets show that the injection of contextual features can greatly improve the accuracy of time series prediction. Comparison with ten frequently used models, including k-nearest neighbor (k-NN), support vector machine (SVM), decision tree (DT), gradient booting decision tree (GBDT), random forest (RF), stacked autoencoder (SAE), and four classic RNNs, also shows the proposed models outperform these state-of-the-art traffic prediction methods in terms of accuracy and stability.

A method to identify wet ball mill’s load based on CEEMDAN, RCMDE and SRNN classification

Ball mill plays a key role in mineral processing plant, and its load identification for optimal control has great significance for the energy consumption reduction and production efficiency improvement. The vibration signal of ball mill shell contains abundant load information, which can be used to identify ball mill load. However, due to the non-linear and non-stationary characteristics of vibration signals, as well as the heavy background noises, the load identification becomes a challenging task in practice. In this paper, a novel method based on complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN), refined composite multi-scale dispersion entropy (RCMDE), and stacked recurrent neural network (SRNN) is proposed. First, CEEMDAN algorithm is used to decompose the ball mill’s vibration signals and obtain the intrinsic mode function (IMF) components. Then, the sensitive IMF components are selected through the correlation coefficient method, and the signal is reconstructed with the sensitive IMF components. Secondly, the RCMDE of the reconstructed signal is calculated to obtain the load feature vector, and the dimension of the feature vector is reduced by principle component analysis (PCA). Thirdly, the SRNN is applied to establish a load recognition model, taking the feature vector as its input and the load state as its output. The experimental results show encouraging accuracy to apply this approach to recognize the wet ball mill’s load, with a recognition rate of 98.67%.

Short-term wind power forecasting by stacked recurrent neural networks with parametric sine activation function

Short-term wind power forecasting is a challenging issue in renewable power generation and distribution due to inevitable intermittency, complex fluctuation and high volatility of wind energy sources. Deep learning methods, especially recurrent neural networks (RNN), have superior performance among the existing methods. This paper proposes a stacked recurrent neural network (SRNN) with parametric sine activation function (PSAF) algorithm for wind power forecasting, whose parametrization can be tuned adaptively and iteratively during prediction. More precisely, firstly, nonlinear difference equation modeling of SRNN-PSAF is contrived, together with back-propagation training parametrization; secondly, sensitivity analysis of the proposed model with respect to forecasting accuracy is conducted, which provides design guidance in practical situations; thirdly, the proposed model is applied to short-term wind power forecasting based on the national renewable energy laboratory (NREL) wind power data; finally, simulation results are collected to illustrate high capability in retrieving manifold features in wind power sources.

2-D Skeleton-Based Action Recognition via Two-Branch Stacked LSTM-RNNs

Action recognition in video sequences is an interesting field for many computer vision applications, including behavior analysis, event recognition, and video surveillance. In this article, a method based on 2D skeleton and two-branch stacked Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) cells is proposed. Unlike 3D skeletons, usually generated by RGB-D cameras, the 2D skeletons adopted in this article are reconstructed starting from RGB video streams, therefore allowing the use of the proposed approach in both indoor and outdoor environments. Moreover, any case of missing skeletal data is managed by exploiting 3D-Convolutional Neural Networks (3D-CNNs). Comparative experiments with several key works on KTH and Weizmann datasets show that the method described in this paper outperforms the current state-of-the-art. Additional experiments on UCF Sports and IXMAS datasets demonstrate the effectiveness of our method in the presence of noisy data and perspective changes, respectively. Further investigations on UCF Sports, HMDB51, UCF101, and Kinetics400 highlight how the combination between the proposed two-branch stacked LSTM and the 3D-CNN-based network can manage missing skeleton information, greatly improving the overall accuracy. Moreover, additional tests on KTH and UCF Sports datasets also show the robustness of our approach in the presence of partial body occlusions. Finally, comparisons on UT-Kinect and NTU-RGB+D datasets show that the accuracy of the proposed method is fully comparable to that of works based on 3D skeletons.

Load Disaggregation Using One-Directional Convolutional Stacked Long Short-Term Memory Recurrent Neural Network

Reliable information about the active loads in the energy system allows for effective and optimized energy management. An important aspect of intelligent energy monitoring system is load disaggregation. The proliferation of direct current (dc) loads has spurred the increasing research interest in extra low voltage (ELV) dc grids. Artificial intelligence, such as deep learning algorithms of stacked recurrent neural network (RNN), improved results on a variety of regression and classification tasks. This paper proposes a 1-D convolutional stacked long short-term memory RNN technique for the bottom-up approach in load disaggregation using single sensor multiple loads ELV dc picogrids. This eliminates the requirement for communication and intelligence on every load in the grid. The proposed technique was applied on two different dc picogrids to test the algorithm's robustness. The proposed technique produced excellent result of over 98% accuracy for smart loads and over 99% accuracy for dumb loads in ELV dc picogrid.

Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks.

This paper presents an anomaly detection method that is based on a sparse coding inspired Deep Neural Networks (DNN). Specifically, in light of the success of sparse coding based anomaly detection, we propose a Temporally-coherent Sparse Coding (TSC), where a temporally-coherent term is used to preserve the similarity between two similar frames. The optimization of sparse coefficients in TSC with the Sequential Iterative Soft-Thresholding Algorithm (SIATA) is equivalent to a special stacked Recurrent Neural Networks (sRNN) architecture. Further, to reduce the computational cost in alternatively updating the dictionary and sparse coefficients in TSC optimization and to alleviate hyperparameters selection in TSC, we stack one more layer on top of the TSC-inspired sRNN to reconstruct the inputs, and arrive at an sRNN-AE. We further improve sRNN-AE in the following aspects: i) rather than using a predefined similarity measurement between two frames, we propose to learn a data-dependent similarity measurement between neighboring frames in sRNN-AE to make it more suitable for anomaly detection; ii) to reduce computational costs in the inference stage, we reduce the depth of the sRNN in sRNN-AE and, consequently, our framework achieves real-time anomaly detection; iii) to improve computational efficiency, we conduct temporal pooling over the appearance features of several consecutive frames for summarizing information temporally, then we feed appearance features and temporally summarized features into a separate sRNN-AE for more robust anomaly detection. To facilitate anomaly detection evaluation, we also build a large-scale anomaly detection dataset which is even larger than the summation of all existing datasets for anomaly detection in terms of both the volume of data and the diversity of scenes. Extensive experiments on both a toy dataset under controlled settings and real datasets demonstrate that our method significantly outperforms existing methods, which validates the effectiveness of our sRNN-AE method for anomaly detection. Codes and data have been released at https://github.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection.

Abstract 448: Predicting Blood Pressure Response to Fluid Bolus Therapy Using Neural Networks with Clinical Interpretability

Background: Fluid bolus therapy (FBT), the rapid infusion of fluid, has been recommended as the primary-line treatment for acute hypotensive episode (AHE) that occurs in about 41% of patients in ICU. However, previous studies have reported that approximately one-third of the acute hypotensive patients do not successfully respond to FBT treatment. Avoiding the administration of FBT that will not successfully resolve AHE might prevent an inappropriate increase of the total fluid volume administered to ICU patients, potentially reducing their risk for severe organ dysfunction and increased mortality. Methods: Our study utilized regression models and attention-based recurrent neural network (RNN) algorithms and two large-scale information system databases, the multi-clinical MIMIC-ICU one and the multi-center Philips eICU CRD one, to predict the successful response to FBT among hypotensive patients in ICUs. We investigated both time-aggregated modeling and time-series modeling using RNN with the attention mechanism (AM) for clinical interpretability. The successful FBT is defined by intensive care experts as the presence of the maximum mean atrial pressure (MAP) > 1.15 * average (MAP) at least once, where maximum(MAP) is the maximal MAP from the FBT starting time to two hours after FBT, and average (MAP) is the average MAP from 30 minutes before FBT until 10 minutes after FBT. Results: The stacked RNN with AM yielded the highest accuracy of 0.852 and area under the curve (AUC) value of 0.925 when trained and tested on the MIMIC-ICU dataset. The top features learned from regression include the patient's respiratory rate, diastolic pressure, temperature, and bicarbonate and base excess levels in blood. Preliminary results from training and testing the RNN on the Philips eICU-CRD database yielded an accuracy of 0.812 and AUC value of 0.769. We were also able to identify timesteps close to the time of FBT administration as clinically meaningful using the RNN models with AM. Conclusion: This is the first study that utilizes machine learning for identifying hypotensive patients in ICUs who will have sufficient blood pressure recovery after FBT. Utilizing AM and identifying the top features learned also provided clinical interpretability to the models we used.

Jump detection in financial time series using machine learning algorithms

In this paper, we develop a new Hybrid method based on machine learning algorithms for jump detection in financial time series. Jump is an important behavior in financial time series, since it implies a change in volatility. Ones can buy the volatility instrument if ones expect the volatility will bloom up in the future. A jump detection model attempts to detect short-term market instability, since it could be jumping up or down, instead of a directional prediction. The directional prediction can be considered as a momentum or trend following, which is not the focus of this paper. A jump detection model is commonly applied in a systematic fast-moving strategy, which reallocates the assets automatically. Also, a systematic opening position protection strategy can be driven by a jump detection model. For example, for a tail risk protection strategy, a pair of long call and put option order could be placed in the same time, in order to protect the open position given a huge change in volatility. One of the key differentiations of the proposed model with the classical methods of time-series anomaly detection is that, jump threshold parameters are not required to be predefined in our proposed model. Also the model is a combination of a Long short-term memory (LSTM) neural network model and a machine learning pattern recognition model. The LSTM model is applied for time series prediction, which predicts the next data point. The historical prediction errors sequence can be used as the information source or input of the jump detection model/module. The machine learning pattern recognition model is applied for jump detection. The combined model attempts to determine whether the current data point is a jump or not. LSTM neural network is a type of Recurrent Neural Networks (RNNs). LSTM records not only the recent market, but also the historical status. A stacked RNN is trained on a dataset which is mixed with normal and anomalous data. We compare the performance of the proposed Hybrid jump detection model and different pattern classification algorithms, such as k-nearest neighbors algorithm identifier, Hampel identifier, and Lee Mykland test. The model is trained and tested using real financial market data, including 11 global stock market in both developed and emerging markets in US, China, Hong Kong, Taiwan, Japan, UK, German, and Israel. The experiment result shows that the proposed Hybrid jump detection model is effective to detect jumps in terms of accuracy, comparing to the other classical jump detection methods.

Human motion recognition exploiting radar with stacked recurrent neural network

We develop a novel radar-based human motion recognition technique that exploits the temporal sequentiality of human motions. The stacked recurrent neural network (RNN) with long short-term memory (LSTM) units is employed to extract sequential features for automatic motion classification. The spectrogram of raw radar data is used as the network input to utilize the time-varying Doppler and micro-Doppler signatures for human motion characterization. Based on experimental data, we verified that a stacked RNN with two 36-cell LSTM layers successfully classifies six different types of human motions.

Intent Prediction of Pedestrians via Motion Trajectories Using Stacked Recurrent Neural Networks

The problem of intent understanding between highly and fully automated vehicles and vulnerable road users (VRUs) such as pedestrians in urban traffic environment has got some momentum over the past few years. Previous work has been tackling the problem using two common approaches, namely dynamical motion modeling and motion planning. In this paper, a novel radical end-to-end data-driven approach is proposed for long-term intent prediction of VRUs in urban traffic environment based solely on their motion trajectories. In the proposed approach, we utilized the widely adopted architecture of recurrent neural networks with Long-Short Term Memory (LSTM) modules to form a deep stacked LSTM network. Three common approaches used in the literature were compared against our proposed approach over two different real-world datasets involving pedestrians collected from vehicle-based stereo cameras. The results over the testing datasets showed that the proposed approach achieved higher accuracies over most of the scenarios of the testing datasets with a small mean lateral position error of 0.48 m. Moreover, the proposed approach showed also a significant generalization capability over totally unobserved testing scenes during the training phase with only 0.58 m in mean lateral position error.

Evaluation of Recurrent Neural Network and its Variants for Intrusion Detection System (IDS)

This article describes how sequential data modeling is a relevant task in Cybersecurity. Sequences are attributed temporal characteristics either explicitly or implicitly. Recurrent neural networks (RNNs) are a subset of artificial neural networks (ANNs) which have appeared as a powerful, principle approach to learn dynamic temporal behaviors in an arbitrary length of large-scale sequence data. Furthermore, stacked recurrent neural networks (S-RNNs) have the potential to learn complex temporal behaviors quickly, including sparse representations. To leverage this, the authors model network traffic as a time series, particularly transmission control protocol / internet protocol (TCP/IP) packets in a predefined time range with a supervised learning method, using millions of known good and bad network connections. To find out the best architecture, the authors complete a comprehensive review of various RNN architectures with its network parameters and network structures. Ideally, as a test bed, they use the existing benchmark Defense Advanced Research Projects Agency / Knowledge Discovery and Data Mining (DARPA) / (KDD) Cup ‘99' intrusion detection (ID) contest data set to show the efficacy of these various RNN architectures. All the experiments of deep learning architectures are run up to 1000 epochs with a learning rate in the range [0.01-0.5] on a GPU-enabled TensorFlow and experiments of traditional machine learning algorithms are done using Scikit-learn. Experiments of families of RNN architecture achieved a low false positive rate in comparison to the traditional machine learning classifiers. The primary reason is that RNN architectures are able to store information for long-term dependencies over time-lags and to adjust with successive connection sequence information. In addition, the effectiveness of RNN architectures are shown for the UNSW-NB15 data set.