Layer Recurrent Neural Network Research Articles

Sentiment prediction based on neural network

Recent years have seen a surge in interest in deep learning, and numerous new neural network models have appeared. In this study, we use CNN and LSTM, two popular deep learning models, to predict sentiment. We use the bidirectional model for the LSTM model. We make predictions by integrating the feature vectors of both sides after receiving sentences inputted in the reverse order from both ends. The recurrent neural network layer is replaced with the CNN model, which defines numerous one-dimensional convolution kernels to conduct convolution operations on the input. The two models are assessed and compared based on how well they anticipate and analyze sentence emotion tones. By comparison, we discover that the BiLSTM model's prediction accuracy on the test set is 10% greater than CNN's, despite requiring more training time.

Congestive Heart Failure Detection From ECG Signals Using Deep Residual Neural Network

The early and accurate detection of congestive heart failure (CHF) using an electrocardiogram (ECG) is of great significance for improving the survival rate of patients. Existing approaches show limited detection accuracy as they fail to capture the temporal ECG dynamics. Also, these methods lack model transparency and are often difficult to interpret. This article proposes a novel end-to-end diagnostic attention-based deep residual recurrent neural network (DA-DRRNet) that effectively captures the temporal dynamics and extracts high-level attentive representations for accurate CHF detection. Specifically, we first employ a recurrent neural network (RNN) layer to encode the temporal dynamics from the raw ECG beats. Then, multilayered RNNs with residual connections are incorporated to extract high-level feature representations hierarchically. The residual connections allow gradients in deep RNN to propagate effectively, thereby improving the network’s representation ability. Finally, an attention module identifies the hidden vectors corresponding to the diagnostically prominent ECG characteristics to form an attentive representation for improved CHF detection. Using ECG signals from the three publicly available datasets (BIDMC-CHF, PTBDB, and MIT-BIH NSRDB), the proposed method achieves an impressive accuracy of 98.57 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> and nearly 100 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\%$</tex-math> </inline-formula> for beat-level and 24-h record-level diagnosis, respectively. Notably, the analysis of learned attention weights demonstrates that the proposed model focuses on the clinically relevant ECG features that characterize CHF. This model transparency and improved detection results advance research in this field and provide a reliable and transparent diagnostic system for CHF analysis.

Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml

Recurrent neural networks have been shown to be effective architectures for many tasks in high energy physics, and thus have been widely adopted. Their use in low-latency environments has, however, been limited as a result of the difficulties of implementing recurrent architectures on field-programmable gate arrays (FPGAs). In this paper we present an implementation of two types of recurrent neural network layers—long short-term memory and gated recurrent unit—within the hls4ml framework. We demonstrate that our implementation is capable of producing effective designs for both small and large models, and can be customized to meet specific design requirements for inference latencies and FPGA resources. We show the performance and synthesized designs for multiple neural networks, many of which are trained specifically for jet identification tasks at the CERN Large Hadron Collider.

Speech Recognition for Air Traffic Control via Feature Learning and End-to-End Training

In this work, we propose a new automatic speech recognition (ASR) system based on feature learning and an end-to-end training procedure for air traffic control (ATC) systems. The proposed model integrates the feature learning block, recurrent neural network (RNN), and connectionist temporal classification loss to build an end-to-end ASR model. Facing the complex environments of ATC speech, instead of the handcrafted features, a learning block is designed to extract informative features from raw waveforms for acoustic modeling. Both the SincNet and 1D convolution blocks are applied to process the raw waveforms, whose outputs are concatenated to the RNN layers for the temporal modeling. Thanks to the ability to learn representations from raw waveforms, the proposed model can be optimized in a complete end-to-end manner, i.e., from waveform to text. Finally, the multilingual issue in the ATC domain is also considered to achieve the ASR task by constructing a combined vocabulary of Chinese characters and English letters. The proposed approach is validated on a multilingual real-world corpus (ATCSpeech), and the experimental results demonstrate that the proposed approach outperforms other baselines, achieving a 6.9% character error rate.

Recorded recurrent deep reinforcement learning guidance laws for intercepting endoatmospheric maneuvering missiles

This work proposes a recorded recurrent twin delayed deep deterministic (RRTD3) policy gradient algorithm to solve the challenge of constructing guidance laws for intercepting endoatmospheric maneuvering missiles with uncertainties and observation noise. The attack-defense engagement scenario is modeled as a partially observable Markov decision process (POMDP). Given the benefits of recurrent neural networks (RNNs) in processing sequence information, an RNN layer is incorporated into the agent’s policy network to alleviate the bottleneck of traditional deep reinforcement learning methods while dealing with POMDPs. The measurements from the interceptor’s seeker during each guidance cycle are combined into one sequence as the input to the policy network since the detection frequency of an interceptor is usually higher than its guidance frequency. During training, the hidden states of the RNN layer in the policy network are recorded to overcome the partially observable problem that this RNN layer causes inside the agent. The training curves show that the proposed RRTD3 successfully enhances data efficiency, training speed, and training stability. The test results confirm the advantages of the RRTD3-based guidance laws over some conventional guidance laws.

Animal behavior classification via deep learning on embedded systems

We develop an end-to-end deep-neural-network-based algorithm for classifying animal behavior using accelerometry data on the embedded system of an artificial intelligence of things (AIoT) device installed in a wearable collar tag. The proposed algorithm jointly performs feature extraction and classification utilizing a set of infinite-impulse-response (IIR) and finite-impulse-response (FIR) filters together with a multilayer perceptron. The utilized IIR and FIR filters can be viewed as specific types of recurrent and convolutional neural network layers, respectively. We evaluate the performance of the proposed algorithm via two real-world datasets collected from total eighteen grazing beef cattle using collar tags. The results show that the proposed algorithm offers good intra- and inter-dataset classification accuracy and outperforms its closest contenders including two state-of-the-art convolutional-neural-network-based time-series classification algorithms, which are significantly more complex. We implement the proposed algorithm on the embedded system of the utilized collar tags’ AIoT device to perform in-situ classification of animal behavior. We achieve real-time in-situ behavior inference from accelerometry data without imposing any strain on the available computational, memory, or energy resources of the embedded system.

Heart sound classification based on equal scale frequency cepstral coefficients and deep learning.

Heart diseases represent a serious medical condition that can be fatal. Therefore, it is critical to investigate the measures of its early prevention. The Mel-scale frequency cepstral coefficients (MFCC) feature has been widely used in the early diagnosis of heart abnormity and achieved promising results. During feature extraction, the Mel-scale triangular overlapping filter set is applied, which makes the frequency response more in line with the human auditory property. However, the frequency of the heart sound signals has no specific relationship with the human auditory system, which may not be suitable for processing of heart sound signals. To overcome this issue and obtain a more objective feature that can better adapt to practical use, in this work, we propose an equal scale frequency cepstral coefficients (EFCC) feature based on replacing the Mel-scale filter set with a set of equally spaced triangular overlapping filters. We further designed classifiers combining convolutional neural network (CNN), recurrent neural network (RNN) and random forest (RF) layers, which can extract both the spatial and temporal information of the input features. We evaluated the proposed algorithm on our database and the PhysioNet Computational Cardiology (CinC) 2016 Challenge Database. Results from ten-fold cross-validation reveal that the EFCC-based features show considerably better performance and robustness than the MFCC-based features on the task of classifying heart sounds from novel patients. Our algorithm can be further used in wearable medical devices to monitor the heart status of patients in real time with high precision, which is of great clinical importance.

Hybrid ensemble framework with self-attention mechanism for social spam detection on imbalanced data

Cybercriminals use social media platforms to disseminate spam, misleading facts, fake news, and malicious links. Blocking such deceptive social media spam is essential. However, extracting relevant features from social networks is challenging due to privacy and time constraints. Traditional frequency-based word representation techniques are time-consuming and inefficient in producing contextual word vectors. Word embeddings and deep learning models have recently shown good results in text classification. Also, most existing approaches assumed balanced class distribution, which is false for most real-world datasets. In this paper, an attempt is made to advance the performance of the social spam detection system by leveraging dataset balancing, advanced word embedding techniques, machine learning, and deep learning approaches with the self-attention mechanism. In the proposed framework, the datasets are balanced using NearMiss and SmoteTomek techniques to feed several machine-learning models. Later, the baseline ML models and proposed voting-based ensemble models are evaluated on imbalanced and balanced datasets. For the proposed deep learning-based hybrid approaches, embeddings are generated using GloVe and FastText word embeddings on the balanced combined dataset and passed into the deep neural network comprised of Conv1D and Bi-directional recurrent neural network layers with the self-attention mechanism for improved context understanding and effective results. This study examines hybrid approaches for detecting social spam using imbalanced social network data and picks the optimum combination. Besides, Machine learning ensembles, word embeddings, deep learning with hyper-parameter optimization, and a self-attention method are compared thoroughly. Experiments and comparisons with other techniques show that the proposed hybrid framework with deep learning-based approaches achieves better performance.

Neural Network Based Approach for Steady-State Stability Assessment of Power Systems

The quest for an intelligence compliance system to solve power stability problems in real-time with high predictive accuracy, and efficiency has led to the discovery of deep learning (DL) techniques. This paper investigates the potency of several artificial neural network (ANN) techniques in assessing the steady-state stability of a power system. The new voltage stability pointer (NVSP) was employed to parameterize and reduce the input data to the neural network algorithms to predict the proximity of power systems to voltage instability. In this study, we consider five neural network algorithms viz. feedforward neural network (FFNN), cascade-forward neural network (CFNN), layer recurrent neural network (LRNN), linear layer neural network (LLNN), and Elman neural network (ENN). The evaluation is based on the predictability and accuracy of these techniques for dynamic stability in power systems. The neural network algorithms were trained to mimic the NVSP dataset using a Levenberg-Marquardt (LM) model. Similarly, the performance analyses of the neural network techniques were deduced from the regression learner algorithm (RLA) using a root-mean-squared error (rmse) and response plot graph. The effectiveness of these NN algorithms was demonstrated on the IEEE 30-bus system and the Nigerian power system. The simulation results show that the FFNN and the CFNN possess a relatively better performance in terms of accuracy and efficiency for the considered power networks.

Cardiac Magnetic Resonance Image Segmentation Method Based on Multi-Scale Feature Fusion and Sequence Relationship Learning.

Accurate segmentation of the left atrial structure using magnetic resonance images provides an important basis for the diagnosis of atrial fibrillation (AF) and its treatment using robotic surgery. In this study, an image segmentation method based on sequence relationship learning and multi-scale feature fusion is proposed for 3D to 2D sequence conversion in cardiac magnetic resonance images and the varying scales of left atrial structures within different slices. Firstly, a convolutional neural network layer with an attention module was designed to extract and fuse contextual information at different scales in the image, to strengthen the target features using the correlation between features in different regions within the image, and to improve the network's ability to distinguish the left atrial structure. Secondly, a recurrent neural network layer oriented to two-dimensional images was designed to capture the correlation of left atrial structures in adjacent slices by simulating the continuous relationship between sequential image slices. Finally, a combined loss function was constructed to reduce the effect of positive and negative sample imbalance and improve model stability. The Dice, IoU, and Hausdorff distance values reached 90.73%, 89.37%, and 4.803 mm, respectively, based on the LASC2013 (left atrial segmentation challenge in 2013) dataset; the corresponding values reached 92.05%, 89.41% and 9.056 mm, respectively, based on the ASC2018 (atrial segmentation challenge at 2018) dataset.

Self-adapting PI controller for grid-connected DFIG wind turbines based on recurrent neural network optimization control under unbalanced grid faults

• Establish the MPPT with optimal speed ratio combined with hidden layer recurrent neural network (HLRNN) to obtain faster convergence. • Optimize the PI controller gains by using a hidden layer recurrent neural network. • Investigate the active power behavior in the presence of unbalanced grid faults. • Amelioration of the response time of the produced power compared with some other existing neural network schemes. This paper suggests the Hidden Layer Recurrent Neural Network (HLRNN) for controlling a wind power generation system based on a dual-fed induction generator. The generator's stator is connected directly to the electrical grid, while the rotor is linked through bidirectional converters. A PI controller-based Indirect Vector Control (IVC) scheme has been established to pilot the system. The PI regulator allows linear systems to perform well, but when subjected to physical variation conditions, the system's response becomes unstable, making the PI controller insufficient. This paper aims to ensure the PI controller gains self-adaptation regardless of the severity of the circumstances. The effectiveness of the proposed control is demonstrated by taking into account critical conditions such as wind speed changes, generator parameter variations, and asymmetrical faults in the grid. Furthermore, it is confirmed by the enhanced performance and the reduced oscillations during a voltage dip. The simulation results, achieved using MATLAB/Simulink, demonstrate that the response time is reduced to 1.8 (m s), the static error is minimized to 0.16%, and the overshoot is improved to 0.24% when compared to the IVC based on the PI controller and some existing neural network schemes.

Temporal Convolutional Networks with RNN approach for chaotic time series prediction

The prediction of chaotic time series, which constitutes many systems in the field of science and engineering, has recently become the focus of attention of researchers. Chaotic time series prediction is making future predictions about these systems using previously observed data for a nonlinear chaotic system with a known initial condition. Chaotic time series prediction can be applied in many fields such as weather forecasting, finance and stock markets. Many disciplines work on solving time series prediction problem, ranging from forecasting weather events days in advance to traders predicting the future of stocks. In recent studies, it has been observed that hybrid deep neural network methods give better performance in solving time series prediction problems and have gained popularity in order to benefit from the advantages of more than one method in solving such problems. In this study, a hybrid deep neural network architecture is proposed for chaotic time series prediction. The used hybrid approach includes both temporal convolutional network to extract low level features from input and recurrent neural network layers such as long short-term memory and gated recurrent units to capture temporal information. Simulations were carried out on nine different chaotic time series dataset which are obtained from Lorenz, Rössler and a Lorenz-like chaotic equation sets, and twenty-one electrocardiogram (ECG) recordings of patients with arrhythmias. In the benchmark study, in which twelve different methods, including classical machine learning, deep neural network and hybrid models were used, the proposed model achieved the best prediction performance with an average root-mean-square error (RMSE) value of 0.0022 for chaotic dataset and 0.0082 for ECG arrhythmia dataset. Performance evaluation metrics show that the proposed hybrid architecture can compete with the models in state-of-the-art studies in chaotic time series prediction.

Predictive modelling of Parkinson’s disease progression based on RNA-Sequence with densely connected deep recurrent neural networks

The advent of recent high throughput sequencing technologies resulted in unexplored big data of genomics and transcriptomics that might help to answer various research questions in Parkinson’s disease (PD) progression. While the literature has revealed various predictive models that use longitudinal clinical data for disease progression, there is no predictive model based on RNA-Sequence data of PD patients. This study investigates how to predict the PD Progression for a patient’s next medical visit by capturing longitudinal temporal patterns in the RNA-Seq data. Data provided by Parkinson Progression Marker Initiative (PPMI) includes 423 PD patients without revealing any race, sex, or age information with a variable number of visits and 34,682 predictor variables for 4 years. We propose a predictive model based on deep Recurrent Neural Network (RNN) with the addition of dense connections and batch normalization into RNN layers. The results show that the proposed architecture can predict PD progression from high dimensional RNA-seq data with a Root Mean Square Error (RMSE) of 6.0 and a rank-order correlation of (r = 0.83, p < 0.0001) between the predicted and actual disease status of PD.

Real-time neural speech enhancement based on temporal refinement network and channel-wise gating methods

Neural speech enhancement systems have seen dramatic improvements in performance recently. However, it is still difficult to create systems that can operate in real-time, with low delay, low complexity, and causality. In this paper, we propose a temporal and channel attention framework for a U-Net-based speech enhancement architecture that uses short analysis frame lengths. Specifically, we propose an attention-based temporal refinement network (TRN) that estimates convolutional features subject to the importance of temporal location. By adding the TRN output to the channel-attentive convolution output, we can further enhance speech-related features even in low-attentive channel outputs. To further improve the representation power of the convolutional features, we also apply a squeeze-and-excitation (SE)-based channel attention mechanism for three different network modules: main convolutional blocks after processing the TRN, skip connections, and residual connections in the bottleneck recurrent neural network (RNN) layer. In particular, a channel-wise gate architecture placed on the skip connections and residual connections reliably controls the data flow, which avoids transferring redundant information to the following stages. We show the effectiveness of the proposed TRN and channel-wise gating methods by visualizing the spectral characteristics of the corresponding features, evaluating overall enhancement performance, and performing ablation studies in various configurations. Our proposed real-time enhancement system outperforms several recent neural enhancement models in terms of quality, model size, and complexity.

Brain Computer Interface Based Thought Recognition System Using a Hybrid Deep Learning Model

ABSTRACT Brain-Computer Interface (BCI) connects the human brain with computers and electronic devices. The signals from human brain are processed using several deep-learning techniques to convert them into a comprehensible form. Among these techniques, the convolutional neural network (CNN) model has excellent performance in BCI recognition. However, the existing CNN model is prone to over-fitting and has limitations with accuracy. The model complexity must be increased to achieve better accuracy. To address these concerns, a novel hybrid R-CNN model for BCI thought recognition is proposed in this work. The convolution layer of CNN and the Long short-term memory (LSTM) layer of recurrent neural network (RNN) is utilized for this purpose. A batch normalization layer is also instigated to reduce over-fitting. Further, a rectified linear unit (ReLU) is engaged to speed up training under as low as five epochs, along with a custom optimizer which optimizes some default values within the optimizer. Experiments are performed with BCI datasets of two different file sizes with different records. The first dataset size is 6.5 MB having 60684 records with three classes, and the second dataset size is 10.1 MB having 94119 records with five classes. Consequently, the proposed hybrid model exhibits a higher average accuracy of 95% for 6.5 MB file size and 98% for 10.1 MB file size, which is superior compared to the accuracy of existing deep learning models. Furthermore, the efficiency of the proposed novel hybrid R-CNN model is evaluated with some other performance measures such as F1-score, recall and precision.

Improved action proposals using fine-grained proposal features with recurrent attention models

Recent models for the temporal action proposal task show that local properties can be an alternative to the region proposal network (RPN) for generating good proposal candidates on untrimmed videos. In this study, we devise an RPN model with a new two-stage pipeline and a new joint scoring function for temporal proposals. The evaluation of local properties is integrated into our RPN model to search for the best proposal candidates that can be distinguished mainly in fine details of proposal regions. Our network models proposals in multiple scales using two recurrent neural network layers with attention mechanisms. We observe that joint training of the RPN with local clues and multi-scale modeling of proposals with recurrent attention mechanisms improve the performance of the proposal generation task. Our model yields state-of-the-art results on the THUMOS-14 and comparable results on the ActivityNet-1.3 datasets.

An adaptive active power rolling dispatch strategy for high proportion of renewable energy based on distributed deep reinforcement learning

In this article, an adaptive active power rolling dispatch strategy based on distributed deep reinforcement learning is proposed to deal with the uncertainty of high-proportioned renewable energy. For each agent, by using recurrent neural network layers and graph attention layers in its network structure, we aim to improve the generalization ability of the multiple agents in active power flow control. Furthermore, a regional graph attention network algorithm, which can effectively help agents aggregate the regional information of their neighborhood, is proposed to improve the information capture ability of agents. We adopt the structure of ‘centralized training, distributed execution’ to help agents improve the effectiveness of proposed methods in dynamic environments. The case studies demonstrate that the proposed algorithm can help multi-agents learn effective active power control strategies. Each agent has a strong generalization ability in terms of time granularity and network topology. We expect that such an approach can improve the practicability and adaptability of distributed AI method on power system control issues.

Batch-Normalized Deep Recurrent Neural Network for High-Speed Nonlinear Circuit Macromodeling

In order to model high-speed nonlinear circuits, recurrent neural network (RNN) has been widely used in computer-aided design (CAD) area to achieve high performance and fast models compared with the existing models. Despite their advantages, they still have challenges such as large training time and limited test accuracy. In this article, the batch normalization (BN) method is applied to deep RNN leading to a much shorter training time and more accurate models compared with the conventional RNN. The proposed BN-RNN method works by modifying the distribution of the internal nodes of a deep network in the training course as an internal auxiliary shift yielding a much faster training. Indeed, the internal covariance shift will be reduced and the training of deep neural networks will be accelerated via a normalization step applied to the layers of RNN. BN-RNN, moreover, has a beneficial effect on gradient flow through the grid by reducing the dependence of gradients on the scale of network parameters or their initial values. This provides a much better learning process without the risk of divergence. For verifying the proposed method, time-domain modeling of three high-speed nonlinear circuits operating at the GHz region is provided. Comparisons of the training and test errors between RNN and BN-RNN, and evaluation time comparisons between transistor level and the BN-RNN-based models for these circuits prove the higher speed of the models obtained from the BN-RNN method. In addition, it is shown that training using the proposed method requires much less CPU time and number of epochs.

Deep transform and metric learning network: Wedding deep dictionary learning and neural network

On account of its many successes in inference tasks and imaging applications, Dictionary Learning (DL) and its related sparse optimization problems have garnered a lot of research interest. In DL area, most solutions are focused on single-layer dictionaries, whose reliance on handcrafted features achieves a somewhat limited performance. With the rapid development of deep learning, improved DL methods called Deep DL (DDL), have been recently proposed an end-to-end flexible inference solution with a much higher performance. The proposed DDL techniques have, however, also fallen short on a number of issues, namely, computational cost and the difficulties in gradient updating and initialization. While a few differential programming solutions have been proposed to speed-up the single-layer DL, none of them could ensure an efficient, scalable, and robust solution for DDL methods. To that end, we propose herein, a novel differentiable programming approach, which yields an efficient, competitive and reliable DDL solution. The novel DDL method jointly learns deep transforms and deep metrics, where each DL layer is theoretically reformulated as a combination of one linear layer and a Recurrent Neural Network (RNN). The RNN is also shown to flexibly account for the layer-associated approximation together with a learnable metric. Additionally, our proposed work unveils new insights into Neural Network (NN) and DDL, bridging the combinations of linear and RNN layers with DDL methods. Extensive experiments on image classification problems are carried out to demonstrate that the proposed method can not only outperform existing DDL several counts including, efficiency, scaling and discrimination, but also achieve better accuracy and increased robustness against adversarial perturbations than CNNs.

Finite-Time Disturbance Observer of Nonlinear Systems

In practical applications, for highly nonlinear systems, how to implement control tasks for dynamic systems with uncertain parameters is still a hot research issue. Aiming at the internal parameter fluctuations and external unknown disturbances in nonlinear system, this paper proposes an adaptive dynamic terminal sliding mode control (ADTSMC) based on a finite-time disturbance observer (FTDO) for nonlinear systems. A finite-time disturbance observer is designed to compensate for the unknown uncertainties and a dynamic terminal sliding mode control (DTSMC) method is developed to achieve finite time convergence and weaken system chattering. Moreover, a dual hidden layer recurrent neural network (DHLRNN) estimator is proposed to approximate the sliding mode gain, so that the switching item gain is not overestimated and optimal value is obtained. Finally, simulation experiments of an active power filter model verify the designed ADTSMC method has better steady-state and dynamic-steady compensation effects with at least 1% THD reduction in the presence of nonlinear load and disturbances compared with the simple adaptive DTSMC law.