Conventional Recurrent Neural Networks Research Articles

Speech Isolation and Recognition in Crowded Noise Using a Dual-Path Recurrent Neural Network

Speech separation is crucial for effective speech processing in multi-talker conditions, especially in real-time, low-latency applications. In this study, the Time-Domain Audio Separation Network (TasNet) and Dual-Path Recurrent Neural Network were used to perform a time-domain multiple-speaker speech separation challenge. One-dimensional conventional recurrent neural networks (RNNs) are not capable of accurately simulating long sequences. When their receptive length exceeds the sequence field, 1-D CNNs cannot recreate utterance-level sequences. Dual-Path Recurrent Neural Network (DPRNN) breaks up the lengthy sequential input that progressively performs intra- and inter-chunk operations with input lengths proportional to the square root of the beginning sequence length. Model outputs are more efficient than earlier systems, improving performance on the Libri Mix dataset. Investigations show that the DPRNN, sample-level modeling, and time-domain audio separation network can replace present methods. EEND-SS and other separation algorithms perform worse than DPRNN. The suggested model was able to achieve (12.376) SI-SDR, (0.969) STOI (short-time objective intelligibility), (12.363) SDR, (9.363) DER, and (97.193) SCA.

A reservoir computing approach in active noise control for vehicle applications

Reservoir Computing (RC) is an alternative approach to Recurrent Neural Networks (RNN) that can avoid some shortcomings of conventional gradient decent-training that are mainly related to high computational complexity and the reduced ability to "learn" long-range dependencies. In RC the input data are transformed into patterns in a high-dimensional space by an RNN called the "reservoir", which remains unchanged during training. The desired output signal is generated as a linear combination of the neuron's signals from the input-excited reservoir. The linear part of the RC architecture, which is called the "readout" can be trained with a simple learning algorithm such as linear regression. In this work an RC architecture is proposed for the attenuation of acoustic disturbances encountered in vehicle cabins such as aircrafts and yachts. At first, a dataset of acoustic pressure measurements from real-world cabins was used to generate the readout and reservoir layers. Subsequent computer simulations demonstrated that the proposed architecture can successfully attenuate such acoustic disturbances. Finally, the performance of RC in Active Noise Cancellation task was compared with both the linear FxLMS algorithm and a conventional RNN, showing that it has several advantages regarding acoustic pressure reduction combined with relatively low computational complexity.

Portfolio Prediction using Deep Learning

This research investigates the use of Long Short-Term Memory (LSTM) networks for predicting stock prices and optimizing investment portfolios. Utilizing a dataset comprising the top 30 U.S. companies by market capitalization from 2009-12-31 to 2021-12-31, excluding AbbVie, Meta, and Tesla due to their later market listings, we demonstrate that LSTM models outperform conventional Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN) in generating accuracy and portfolio returns. By efficiently capturing long-term dependencies within stock price data, LSTMs offer more reliable predictions, which are crucial for optimizing investment portfolios. Our methodology involves data collection and preprocessing, model building using LSTM architecture, and evaluating performance using metrics such as mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), mean absolute percentage error (MAPE), and mean percentage error (MPE). The results indicate that LSTM models not only enhance prediction accuracy but also improve portfolio returns compared to equally weighted and market capitalization weighted portfolios. This research provides significant insights into the benefits of using advanced deep learning techniques like LSTMs for financial market predictions and portfolio management.

Volleyball training video classification description using the BiLSTM fusion attention mechanism

This study aims to explore methods for classifying and describing volleyball training videos using deep learning techniques. By developing an innovative model that integrates Bi-directional Long Short-Term Memory (BiLSTM) and attention mechanisms, referred to BiLSTM-Multimodal Attention Fusion Temporal Classification (BiLSTM-MAFTC), the study enhances the accuracy and efficiency of volleyball video content analysis. Initially, the model encodes features from various modalities into feature vectors, capturing different types of information such as positional and modal data. The BiLSTM network is then used to model multi-modal temporal information, while spatial and channel attention mechanisms are incorporated to form a dual-attention module. This module establishes correlations between different modality features, extracting valuable information from each modality and uncovering complementary information across modalities. Extensive experiments validate the method's effectiveness and state-of-the-art performance. Compared to conventional recurrent neural network algorithms, the model achieves recognition accuracies exceeding 95 % under Top-1 and Top-5 metrics for action recognition, with a recognition speed of 0.04 s per video. The study demonstrates that the model can effectively process and analyze multimodal temporal information, including athlete movements, positional relationships on the court, and ball trajectories. Consequently, precise classification and description of volleyball training videos are achieved. This advancement significantly enhances the efficiency of coaches and athletes in volleyball training and provides valuable insights for broader sports video analysis research.

Explicit Context Integrated Recurrent Neural Network for applications in smart environments

The development and progress in sensor, communication, and computing technologies have led to smart environments. In such environments, data can easily be acquired not only from the monitored entities but also from the surroundings where the entity is operating. The data from the operating environment, which is not useful for learning models in isolation, is referred to as explicit context here. The performance of the predictive models can potentially improve when explicit contexts are taken into account. Typically, the data from various sensors are present in the form of time series. Recurrent Neural Networks (RNNs) are used for such data as they implicitly can deal with temporal contexts. However, while using RNNs for various smart environment applications, the available explicit contexts are often ignored or incorporated as primary (or ordinary) features into the model. Further, the conventional RNN models such as Elman RNN, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU) in their current form do not provide any mechanism to integrate explicit contexts. In this paper, we propose a Context Integrated RNN (CiRNN) that enables integrating explicit contexts represented in the form of contextual features. In CiRNN, the network weights are influenced by contextual features in such a way that the primary input features receive more weight if they are relevant in a given context. To show the efficacy of CiRNN, we selected three application domains, remaining useful life estimation for predictive maintenance, traffic forecasting for intelligent transportation systems, and appliance energy usage prediction for smart homes. These applications typically capture data from various sensors and provide scope for exploiting contextual information. For all three case studies, we used public datasets. The test results show that CiRNN performs better when compared to baseline models that incorporate contextual features as primary features, as well as in comparison to state-of-the-art models. The performance is measured in terms of RMSE, MAE, and score from an asymmetric scoring function. The latter measure is specific to the task of RUL estimation.

Time series data recovery in SHM of large-scale bridges: Leveraging GAN and Bi-LSTM networks

This paper proposes the utilization of a Bidirectional Long Short-Term Memory (Bi-LSTM) network and a Generative Adversarial Network (GAN) model, to recover measured time-series data from Structural Health Monitoring (SHM) systems. In civil engineering, time-series data plays a crucial role in SHM systems. However, unforeseen incidents, such as equipment malfunctions, flawed data collection procedures, or human errors may result in missing data or adversely affect the accuracy of the data collected. To address this challenge, researchers have recently introduced various methods for time series data imputation. Nevertheless, it is evident that many existing approaches are hindered by inherent limitations: (1) neglecting bidirectional temporal correlations; (2) failing to model correlations among variables; (3) generating data that inadequately reflects the distribution of the original dataset, leading to inaccuracies in the recovered data. To overcome these limitations, this study proposes the utilization of Bi-LSTM-GAN for recovering time-series data in the context of SHM. Bi-LSTM demonstrates remarkable proficiency in capturing bidirectional temporal and cross-variable correlations, while GAN is utilized to precisely acquire the distribution of the original data. Additionally, Bi-LSTM significantly bolsters the capacity for long-term data recovery in contrast to a conventional Recurrent Neural Network (RNN). To evaluate the efficacy of this approach, we employ two real-world models: a laboratory-based cable-stayed bridge and an actual three-span continuous bridge. The obtained results compellingly demonstrate that Bi-LSTM-GAN not only achieves data restoration effectively but also yields superior accuracy when compared to conventional GAN model.

Universal approximation property of stochastic configuration networks for time series

For the purpose of processing sequential data, such as time series, and addressing the challenge of manually tuning the architecture of traditional recurrent neural networks (RNNs), this paper introduces a novel approach-the Recurrent Stochastic Configuration Network (RSCN). This network is constructed based on the random incremental algorithm of stochastic configuration networks. Leveraging the foundational structure of recurrent neural networks, our learning model commences with a modest-scale recurrent neural network featuring a single hidden layer and a solitary hidden node. Subsequently, the node parameters of the hidden layer undergo incremental augmentation through a random configuration process, with corresponding weights assigned structurally. This iterative expansion continues until the network satisfies predefined termination criteria. Noteworthy is the adaptability of this algorithm to handle time series data, exhibiting superior performance compared to traditional recurrent neural networks with similar architectures. The experimental results presented in this paper underscore the efficacy of the proposed RSCN for sequence data processing, showcasing its advantages over conventional recurrent neural networks in the context of the performed experiments.

Emotion detection from handwriting and drawing samples using an attention-based transformer model.

Emotion detection (ED) involves the identification and understanding of an individual's emotional state through various cues such as facial expressions, voice tones, physiological changes, and behavioral patterns. In this context, behavioral analysis is employed to observe actions and behaviors for emotional interpretation. This work specifically employs behavioral metrics like drawing and handwriting to determine a person's emotional state, recognizing these actions as physical functions integrating motor and cognitive processes. The study proposes an attention-based transformer model as an innovative approach to identify emotions from handwriting and drawing samples, thereby advancing the capabilities of ED into the domains of fine motor skills and artistic expression. The initial data obtained provides a set of points that correspond to the handwriting or drawing strokes. Each stroke point is subsequently delivered to the attention-based transformer model, which embeds it into a high-dimensional vector space. The model builds a prediction about the emotional state of the person who generated the sample by integrating the most important components and patterns in the input sequence using self-attentional processes. The proposed approach possesses a distinct advantage in its enhanced capacity to capture long-range correlations compared to conventional recurrent neural networks (RNN). This characteristic makes it particularly well-suited for the precise identification of emotions from samples of handwriting and drawings, signifying a notable advancement in the field of emotion detection. The proposed method produced cutting-edge outcomes of 92.64% on the benchmark dataset known as EMOTHAW (Emotion Recognition via Handwriting and Drawing).

Model predictive control of nonlinear processes using transfer learning-based recurrent neural networks

Artificial neural networks (ANNs), one of the deep learning techniques that has sparked a lot of attention recently for its exceptional modeling capabilities of nonlinear systems, are an essential candidate for model-based control systems. In particular, recurrent neural networks (RNN) have shown remarkable capacity to forecast the dynamic behavior of nonlinear processes using time-series data. In order to regulate the rate at which chemical species are produced in intricate processes, RNNs have successfully served as the predictive model in model-based controllers. However, the necessity for massive amounts of data to capture complex processes is a drawback of typical RNN models. With the goal of efficient utilization of the available data, a different, transfer learning-based training approach for RNNs is presented in this work. The transfer learning strategy is taken into consideration to overcome the challenges stemming from lack of data for the construction of RNN models. In particular, weight-sharing RNNs are developed using a priori physical knowledge. Next, a comprehensive analysis of a complex chemical process on a large scale is conducted to showcase the benefits of the suggested approach across various data sizes and its effectiveness when combined with MPC in contrast to conventional RNNs.

Variational Monte Carlo with large patched transformers

Large language models, like transformers, have recently demonstrated immense powers in text and image generation. This success is driven by the ability to capture long-range correlations between elements in a sequence. The same feature makes the transformer a powerful wavefunction ansatz that addresses the challenge of describing correlations in simulations of qubit systems. Here we consider two-dimensional Rydberg atom arrays to demonstrate that transformers reach higher accuracies than conventional recurrent neural networks for variational ground state searches. We further introduce large, patched transformer models, which consider a sequence of large atom patches, and show that this architecture significantly accelerates the simulations. The proposed architectures reconstruct ground states with accuracies beyond state-of-the-art quantum Monte Carlo methods, allowing for the study of large Rydberg systems in different phases of matter and at phase transitions. Our high-accuracy ground state representations at reasonable computational costs promise new insights into general large-scale quantum many-body systems.

Using LSTM neural network to generate music

Music generation is a cutting edge and useful research filed, which is helpful for artists to compose novel melodies as well as revealing potential patterns of music. Recurrent neural network (RNN) is a member of the neural network family, which is commonly used for processing sequential data. It can deal with sequential changes in data compared to normal neural networks. Long short-term memory (LSTM) aims at improving the conventional RNN. It is designed to alleviate the deficiencies of gradient disappearance and gradient explosion that possibly happened in RNN during training. In simple terms, LSTM is superior at grasping long term information than normal RNN. It can record the information that requires to be recorded for a long time and abandon these unimportant features. Unlike RNN, which have merely one way of stacking long-term information. It's quite useful for tasks that require long range dependence. In this work the effectiveness of the LSTM is validated on the music generation task.

Theoretical Foundation and Design Guideline for Reservoir Computing-Based MIMO-OFDM Symbol Detection

In this paper, we derive a theoretical upper bound on the generalization error of reservoir computing (RC), a special category of recurrent neural networks (RNNs). The specific RC implementation considered in this paper is the echo state network (ESN), and an upper bound on its generalization error is derived via the empirical Rademacher complexity (ERC) approach. While recent work in deriving risk bounds for RC frameworks makes use of a non-standard ERC measure and a direct application of its definition, our work uses the standard ERC measure and tools allowing fair comparison with conventional RNNs. The derived result shows that the generalization error bound obtained for ESNs is tighter than the existing bound for vanilla RNNs, suggesting easier generalization for ESNs. With the ESN applied to symbol detection in MIMO-OFDM (Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing) systems, we show how the derived generalization error bound can guide underlying system design. Specifically, the derived bound together with the empirically characterized training loss is utilized to identify the optimum reservoir size in neurons for the ESN-based symbol detector. Finally, we corroborate our theoretical findings with results from simulations that employ 3GPP standards-compliant wireless channels, signifying the practical relevance of our work.

A New Macromodeling Method Based on Deep Gated Recurrent Unit Regularized With Gaussian Dropout for Nonlinear Circuits

In this paper, for the first time, the deep gated recurrent unit (Deep GRU) is used as a new macromodeling approach for nonlinear circuits. Similar to Long Short-Term Memory (LSTM), the GRU has gating units that control the information flow and makes the network less prone to the vanishing gradient problem. Having a smaller number of gates causes GRU to have fewer parameters compared to LSTM leading to better model accuracy. Using the gates leads gradient formulations to have additive nature which helps them to be more resistant to vanishing and consequently learn long sequences of data. The proposed macromodeling method is capable of modeling nonlinear circuits more accurately and using fewer parameters compared to the conventional LSTM macromodeling method. To further improve the GRU performance, a regularization technique called Gaussian dropout is applied in this paper on deep GRU (GDGRU) to reduce the overfitting problem resulting in better test error. Additionally, the models obtained from the proposed techniques are remarkably faster than the original transistor-level models. To verify the superiority of the proposed method, time-domain modeling of three nonlinear circuits is provided. For these circuits, the comparisons of the accuracy and speed between the conventional recurrent neural network (RNN), the LSTM, and the proposed macromodeling methods are provided.

A Multi-Level Attentive Context-Aware Trajectory Prediction Algorithm for Mobile Social Users

The prediction of a user’s trajectory is a key problem in mobility prediction, which has been applied to a range of fields such as location-based service recommendations and traffic planning. The impact of users’ social contacts on mobility is not adequately considered in the current trajectory prediction research. Furthermore, the spatial–temporal dependence of long trajectories is difficult to characterize by conventional recurrent neural network models. A multi-level attentive context-aware trajectory prediction model (MACTP) for mobile social users is proposed in this research to address the above problems. Specifically, users’ social preferences are captured by friend-level attention, and different friends are allocated varying weights. The impact of other check-in points in the trajectory on the present check-in point is considered through check-in-level attention. Trajectory-level attention is used to obtain the representation of historical trajectories influenced by current trajectories, as well as the spatial–temporal dependencies of longer trajectories. Experimental results on two real-world datasets demonstrate that the proposed model significantly improves trajectory prediction performance.

Prediction of minimum horizontal stress in oil wells using recurrent neural networks

The calculation of the minimum horizontal stress (Shmin) is critical for well planning and hydraulic fracture design. The in-situ Shmin can be estimated via borehole injection tests or theoretical approaches. However, these approaches are complex, costly, need unavailable tectonic stress data, and can only be performed at a specified depth. To that end, this paper intends to apply the most recent varieties of recurrent neural networks (RNNs), such as conventional RNN, long-short-term memory (LSTM), and gated recurrent unit (GRU), to Shmin time-series prediction for the first time. In the models, 13,956 datasets including six input parameters effective on the Shmin from an oil well in Iran were used. 80 percent of the data (from depth 1936 m to depth 3637 m) was used for training, while 20 percent (from depth 3637 m to depth 4068 m) was used for testing. All of its hyper-parameters were extensively adjusted to maximize the accuracy of the RNN models. The performance of the RNN models was compared to that of six other machine learning approaches using various statistical criteria. All of the models demonstrated potential capacity to forecast Shmin. However, for Shmin prediction in oil wells, the GRU model with 700 epochs, 32 hidden neurons, 8 batch sizes, 3 hidden layers, ReLU activation function, 6 time series length, Nadam optimization algorithm, and 0.5 dropout rate was recommended. It is possible to reduce significantly the time and costs associated with measuring the Shmin in this manner.

Tension: A Python package for FORCE learning.

First-Order, Reduced and Controlled Error (FORCE) learning and its variants are widely used to train chaotic recurrent neural networks (RNNs), and outperform gradient methods on certain tasks. However, there is currently no standard software framework for FORCE learning. We present tension, an object-oriented, open-source Python package that implements a TensorFlow / Keras API for FORCE. We show how rate networks, spiking networks, and networks constrained by biological data can all be trained using a shared, easily extensible high-level API. With the same resources, our implementation outperforms a conventional RNN in loss and published FORCE implementations in runtime. Our work here makes FORCE training chaotic RNNs accessible and simple to iterate, and facilitates modeling of how behaviors of interest emerge from neural dynamics.

Batch-Normalized Deep Recurrent Neural Network for High-Speed Nonlinear Circuit Macromodeling

In order to model high-speed nonlinear circuits, recurrent neural network (RNN) has been widely used in computer-aided design (CAD) area to achieve high performance and fast models compared with the existing models. Despite their advantages, they still have challenges such as large training time and limited test accuracy. In this article, the batch normalization (BN) method is applied to deep RNN leading to a much shorter training time and more accurate models compared with the conventional RNN. The proposed BN-RNN method works by modifying the distribution of the internal nodes of a deep network in the training course as an internal auxiliary shift yielding a much faster training. Indeed, the internal covariance shift will be reduced and the training of deep neural networks will be accelerated via a normalization step applied to the layers of RNN. BN-RNN, moreover, has a beneficial effect on gradient flow through the grid by reducing the dependence of gradients on the scale of network parameters or their initial values. This provides a much better learning process without the risk of divergence. For verifying the proposed method, time-domain modeling of three high-speed nonlinear circuits operating at the GHz region is provided. Comparisons of the training and test errors between RNN and BN-RNN, and evaluation time comparisons between transistor level and the BN-RNN-based models for these circuits prove the higher speed of the models obtained from the BN-RNN method. In addition, it is shown that training using the proposed method requires much less CPU time and number of epochs.

SPIDER: A shallow PCA based network intrusion detection system with enhanced recurrent neural networks

In recent years, ensuring the stability of the huge volume of data generated by billions of users has become the main concern in the field of cyber security. Network security is an extensive part of the cyber security system where intrusions are the leading threat in its way of defense. As a countermeasure to such intrusions in the network, Intrusion Detection Systems (IDSs) have played a major role over the years. Although countless works have already been carried out in this field, most of these works have lagged at some points, considering the scope of exploration through recent extensions. So in this paper, a network anomaly detection model, SPIDER, has been proposed. The SPIDER model combines four updated versions of conventional Recurrent Neural Networks (RNNs), namely Bi-LSTM (Bidirectional Long Short Term Memory), LSTM (Long Short Term Memory), Bi-GRU (Bidirectional Gated Recurrent Unit), and GRU (Gated Recurrent Unit). To deal with the dimensionality problems, Principal Component Analysis (PCA) has been adapted to reduce the data dimensions. The performance of the proposed SPIDER model has been evaluated using the well-known NSL-KDD and UNSW-NB15 datasets to ensure its robustness. When compared to the corresponding values of different models, the proposed model shows a significant improvement in detecting intrusions over existing models.

Time-Series Deep Learning Models for Reservoir Scheduling Problems Based on LSTM and Wavelet Transformation

In 2022, as a result of the historically exceptional high temperatures that have been observed this summer in several parts of China, particularly in the province of Sichuan, residential demand for energy has increased. Up to 70% of Sichuan’s electricity comes from hydropower, thus creating a sensible and practical reservoir scheduling plan is essential to maximizing reservoir power generating efficiency. However, classical optimization, such as back propagation (BP) neural network, does not take into account the correlation of samples in time while generating reservoir scheduling rules. We proposed a prediction model based on LSTM neural network coupled with wavelet transformation (WT-LSTM) to address the problem. In order to extract the reservoir scheduling rules, this paper first gathers the scheduling operation data from the Xiluodu hydropower station and creates a dataset. Next, it uses the feature of the time-series prediction model with the realization of a complex nonlinear mapping function, time-series learning capability, and high prediction accuracy. The results demonstrate that the time-series deep learning network has high learning capability for reservoir scheduling by comparing evaluation indexes such as root mean square error (RMSE), rank-sum ratio (RSR), and Nash–Sutcliffe efficiency (NSE). The WT-LSTM network model put forward in this research offers better prediction accuracy than conventional recurrent neural networks and serves as a reference base for scheduling decisions by learning previous scheduling data to produce outflow solutions, which has some theoretical and practical benefits.

Remaining useful life (RUL) regression using Long–Short Term Memory (LSTM) networks

The accurate prediction of the remaining useful life (RUL) of components is a major concern in electronic circuits. The RUL-based health diagnostics plays an important role in the determination of time-of-failure of a device as an early warning in industrial applications. In this paper, a Long Short Term Memory (LSTM) based regression model is proposed for the prediction of RUL of a Ring Oscillator (RO) circuit utilizing the most essential extracted electrical features of the device. LSTM networks are capable of capturing the temporal dependencies in the time-series data and eliminating the vanishing gradient problem encountered in the conventional recurrent neural networks (RNNs). From Cadence simulations, utilizing the 22 nm CMOS technology library, it has been demonstrated that the RO frequency degradation essentially depends on three major factors including the working temperature, voltage, and most importantly, the device aging parameter. The results show that more than 90% of the cases of the RUL prediction for the 13 and 21 stage constrained under the supply voltage variation from 0.7 V to 0.9 V with the least prediction deviation of 2 days to 6 days.