Layer Recurrent Neural Network Research Articles

Unravelling the black box of machine learning for atrial fibrillation forecast: role of heart rate variability and of premature beats

Abstract Background The identification of patients still in sinus rhythm who will present one month later an atrial fibrillation episode is possible using machine learning (ML) techniques. However, these new ML algorithms do not provide any relevant information about the underlying pathophysiology. Purpose To compare the predictive performance for forecasting AF between a machine learning algorithm and other parameters whose pathophysiological mechanisms are known to play a role in the triggering of arrhythmias (i.e. the count of premature beats (PB) and heart rate variability (HRV) parameters) Material and methods We conducted a retrospective study from an outpatient clinic. 10484 Holter ECG recordings were screened. 250 analysable AF onsets were labelled. We developed a deep neural network model composed of convolutional neural network layers and bidirectional gated recurrent units as recurrent neural network layers that was trained for the forecast of paroxysmal AF episodes, using RR intervals variations. This model works like a black box. For comparison purposes, we used a “random forest” (RF) model of ML to obtain forecast results using HRV parameters with and without PB. This model allows the evaluation of the relevance of HRV parameters and of PB used for the forecast. We calculated the area under the curve of the receiving operating characteristic curve for the different time windows counted in RR intervals before the AF onset. Results As shown in the table, the forecasting value of the deep neural network model (ML) was not superior to the random forest algorithm. Prediction value of both decreased when analyzing the RR intervals further away from the onset of AF Conclusions These results suggest that HRV plays a predominant role in triggering AF episodes and that premature beats could add minor information. Moreover, the closer the window from AF onset, the better the accuracy, regardless of the method used. Such detection algorithms once implemented in pacemakers, might prove useful to prevent AF onset by changing pacing sequence while patients would still be in sinus rhythm, however this remains to be demonstrated Funding Acknowledgement Type of funding source: None

Machine learning applied to anthropogenic seismic events detection in Lai Chau reservoir area, Vietnam

Automatic detection of seismic events is a useful tool for routine data processing. Effective detection saves time and effort in phase picking and events’ location, especially in areas with moderate seismicity at regional and local scales. The Lai Chau area in northern Vietnam is a good example of such a region. An additional difficulty in detection is the anthropogenic origin of reservoir-triggered seismicity observed in this region, where seismicity is non-stationary and there was no prior seismic activity. Neural network event detection was prepared to aid event identifications and further processing of seismic data.An automatic detection system was utilized to reduce the effort of manual interpretation of seismic signals in the region of the Lai Chau dam in North Vietnam while maintaining the detection of weak events at the same level. For this reason, a Single Layer Recurrent Neural Network (SLRNN) was applied. Compared to deep learning algorithms, fewer examples were needed to train the SLRNN.This paper presents a modified version of SLRNN, which additionally uses polarization analysis and the multi-stage learning process. In the first stage, the training data consists of events detected manually and disturbances selected visually by the operator. In the next stages, the earlier trained detection is validated in the successive recording periods. False detections together with new seismic events are added to the training set and the detection is retrained. The multi-stage process significantly reduces false detections.The software allows SLRNN to be used for routine seismic data processing.

Real-time single-channel deep neural network-based speech enhancement on edge devices.

In this paper, we present a deep neural network architecture comprising of both convolutional neural network (CNN) and recurrent neural network (RNN) layers for real-time single-channel speech enhancement (SE). The proposed neural network model focuses on enhancing the noisy speech magnitude spectrum on a frame-by-frame process. The developed model is implemented on the smartphone (edge device), to demonstrate the real-time usability of the proposed method. Perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) test results are used to compare the proposed algorithm to previously published conventional and deep learning-based SE methods. Subjective ratings show the performance improvement of the proposed model over the other baseline SE methods.

Associating Unemployment with Panic Attack Using Deep Learning during COVID-19

Corona Virus Infectious Disease (COVID-19) is newly emerging infectious disease. It is known to the world in late 2019. Due to this, the mental health of employees is disturbed. There is always a fear of unemployment amongst employees due to the present scenario of lockdown. This may even create a panic attack. It has been happening rapidly during COVID-19. It has a great effect on human health. This paper analyses multiple factors that have an impact on causing panic attacks. Deep Learning techniques are explored which detects panic disorders on people. Recurrent Neural Network (RNN) based deep learning framework is utilized in this paper that assembles multiple RNN layers along with other hyper-parameters into a single model. This method is implemented by capturing interfering factors and predicts the panic attack tendency of people during COVID-19. Early prediction of panic attacks may assist in saving life from unwanted circumstances. It is also observed that comparative study between MLP and stacked-RNN classifier indicates significantly better results of proposed model over MLP classifier in terms of evaluating metrics.

Sound event aware environmental sound segmentation with Mask U-Net

This paper proposes an environmental sound segmentation method using Mask U-Net. In recent years, human–robot interactions, especially speech dialogue, have been assessed by auditory scene analysis. Methods, such as noise reduction, section detection, and sound source separation have been proposed for robot audition, acoustic signal processing, and machine learning. However, such conventional approaches have three drawbacks: (1) Many studies have analyzed individual functions, which are regarded as being a cascade. Cascade systems can, however, result in the accumulation of errors generated at each functional block. (2) Unlike conventional cascade systems, deep learning-based methods that simultaneously detect sections, separate sound sources, and identify classes have also been proposed for speech separation. These techniques can be extended for multiple classes of environmental sounds, but their performance becomes degraded with large variations in sound event lengths among classes. (3) In addition, these methods have recurrent neural network layers making it difficult to process calculations in parallel. This paper proposes an environmental sound segmentation method called Mask U-Net, which robustly differentiates sound event lengths among classes. Simulation experiments using a developed 75-class environmental sound data set showed that the proposed method was faster than conventional methods and showed high segmentation performance.

Fuzzy Double Hidden Layer Recurrent Neural Terminal Sliding Mode Control of Single-Phase Active Power Filter

This article focuses on the design of a terminal sliding mode control (TSMC) using a fuzzy double hidden layer recurrent neural network (FDHLRNN) strategy for a single-phase active power filter (APF). A TSMC is proposed to make the tracking error of the system converge to zero in a finite time. An FDHLRNN is proposed and applied in harmonic suppression to approximate the equivalent control and eliminate the unknown disturbance, reducing the role of symbol switching items. The main function of the FDHLRNN is to improve control accuracy and reduce the current distortion rate of the APF. The designed FDHLRNN is the weighted sum of the fuzzy network and the double hidden layer network, having a strong global learning ability. The adaptive parameters of the FDHLRNN are derived by the Lyapunov function to ensure the asymptotic stability of the system. The compensation performance and effectiveness of the proposed TSMC using FDHLRNN strategy are verified by real-time experiments.

Prognostics in Aeronautics with Deep Recurrent Neural Networks

Recurrent neural networks (RNNs) such as LSTM and GRU are not new to the field of prognostics. However, the performance of neural networks strongly depends on their architectural structure. In this work, we investigate a hybrid network architecture that is a combination of recurrent and feed-forward (conditional) layers. Two networks, one recurrent and another feed-forward, are chained together, with inference and weight gradients being learned using the standard back-propagation learning procedure. To better tune the network, instead of using raw sensor data, we do some preprocessing on the data, using mostly simple but effective statistics (researched in previous work). This helps the feature extraction phase and eases the problem of finding a suitable network configuration among the immense set of possible ones. This is not the first proposal of a hybrid network in prognostics but our work is novel in the sense that it performs a more comprehensive comparison of this type of architecture for different RNN layers and number of layers. Also, we compare our work with other classical machine learning methods. Evaluation is performed on two real-world case studies from the aero-engine industry: one involving a critical valve subsystem of the jet engine and another the whole reliability of the jet engine. Our goal here is to compare two cases contrasting micro (valve) and macro (whole engine) prognostics. Our results indicate that the performance of the LSTM and GRU deep networks are significantly better than that of other models.

Integrating layered recurrent ANN with robust control strategy for diverse operating conditions of AGC of the power system

This study presents the structural, operational and control aspects of doubly fed induction generator (DFIG) based wind integrated power systems. The automatic generation control (AGC) of a meshed power system including DFIG-based wind turbines has been framed and investigations under various system perturbation are presented. The two-area system consisting of non-reheat thermal turbines with DFIG and interconnected through parallel AC/DC tie-lines is considered for the study. The system non-idealities such as governor lag and generation rate constraints are taken into consideration. An AGC strategy using a layered recurrent artificial neural network (ANN) is proposed in this work. The gains of the proposed AGC are obtained by effectively training the ANN using a set of reliable data obtained from a widespread range of operating system conditions using robust control strategy. The study also incorporates the design of AGC for the power system using the fuzzy logic concept and other AGC actions such as integral (I), proportional–integral (PI) and proportional–integral–derivative (PID) calculated via the means of particle swarm optimization (PSO). The results obtained with the proposed ANN created AGC are linked and demonstrated their superiority over fuzzy logic PI and traditional PSO-based I/PI/PID AGC strategies under numerous system operating conditions.

A stateless deep learning framework to predict net asset value

Recurrent neural networks (RNN) such as Long Short-Term Memory and Gated Recurrent Unit have recently emerged as a state-of-art neural network architectures to process sequential data efficiently. Thereby, they can be used to model prediction of time series data, since time series values are also a sequence of discrete time data. However, due to various existing RNN architectures, their functioning modes, the growth of hyper parameters and some other bottlenecks arise when it comes to train them. Thus, it becomes perplex to find out the most suited model for a given task. To address these matters, we propose a step-wise approach to predict the time series data, especially net asset value in this paper. We have started the study with the memory size of RNN to set the optimal memory size based on the prediction accuracy. Then, the study follows by analyzing existing data preparation methods and proposing a new one. The proposed data preparation methods prove their effectiveness in both stateless and stateful mode with single RNN layer. Finally, we confront the single RNN layer to stacked and bidirectional RNN to sort out the best performing models based on their prediction accuracy in various time horizons.

GPR-Based EMI Prediction for UAV's Dynamic Datalink

The datalink of an unmanned aerial vehicle (UAV) is vulnerable to external electromagnetic interference (EMI). The characteristic parameters of the EMI signal can be definitively detected by a developed monitoring platform mounted on the UAV, but it is impossible to determine the lost-link thresholds of the datalink in any state by experiment because the state of the datalink varies dynamically with the flight distance and the UAV's attitudes. In this case, whether a certain state of the UAV's datalink will be interrupted or not by the EMI cannot be evaluated. In this article, a method of EMI prediction for the UAV's dynamic datalink based on the Gaussian process regression (GPR) is proposed, two indicators, including the interfering frequency and the operation signal power of the datalink, are taken as the training inputs, and the lost-link thresholds of the datalink obtained by the EMI injection test are taken as the training targets. The results show that this method can effectively fit the training samples and determine a 95% confidence interval of the predictive outputs, which has a low predictive error, less than 2.2 dB. Compared with the methods of the layer recurrent neural network and support vector machine, the GPR method has higher predictive accuracy and a stronger generalization ability. Therefore, the detected EMI characteristic parameters can be used as new input samples to predict the lost-link thresholds in any condition. On this basis, the EMI margin can be calculated to evaluate the degree of the anti-EMI redundancy of the UAV's datalink, which can be used for the active adaptation of the UAV's datalink to EMI in the future.

An End-to-End Model Based on Multiple Neural Networks with Data Augmentation for Keyword Spotting

In this paper, we propose a network for small footprint keyword spotting. It includes four parts, data augmentation, Time-Delay Neural Network (TDNN) and Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and attention mechanism. Data augmentation is Google SpecAugment with time warping and frequency mask and time mask to the spectrum on clean data set and noisy data set. TDNN and CNN model the spectrogram features from the time and space dimensions. RNN-type networks include RNN, Long Short-Term Memory network (LSTM), Gated Recurrent Unit (GRU), Bidirectional Long Short-Term Memory network (BiLSTM), and Bidirectional Gated Recurrent Unit (BiGRU). The RNN extracts hidden layer features and transforms them into high-level representations. The attention mechanism is selected to generate different weights and multiplied by the high-level representation generated by the RNN to obtain a fixed-length vector. Finally, we use a linear transformation and softmax function to generate scores. We also explored the size of attention mechanism, two attention mechanisms, rectified linear unit and hidden layer of RNN. Our model has achieved a true positive rate of 99.81% at a 5% false positive rate.

Multi-source social media data sentiment analysis using bidirectional recurrent convolutional neural networks

Subjectivity detection in the text is essential for sentiment analysis, which requires many techniques to perceive unanticipated means of communication. Few accomplishments adapted to capture the syntactic, semantic, and contextual sentimental information via distributed word representations (DWRs)11Distributed work representations .. This paper, concatenating the DWRs through a weighted mechanism on Recurrent Neural Network (RNN) variants joint with Convolutional Neural network (CNN) distinctively involving weighted attentive pooling (WAP)22Weighted Attentive Pooling .. Whereas, CNNs with traditional pooling operations comprise many layers merely able to capture enough features. Our considerations empower the sentiment analysis over DWRs contains Word2vec, FastText, and GloVe to produce dense efficient concatenated representation (DECR)33Dense Efficient Concatenated Representation . to hold long term dependencies on a single RNN layer acquired by Parts of Speech Tagging (POS) explicitly with verbs, adverbs, and noun only. Then use these representations gained in a way, inputted to CNN contain single convolution layer engaging WAP on multi-source social media data to handle the issues of syntactic and semantic regularities as well as out of vocabulary (OOV) words. Experimentations demonstrate that DWRs together with proposed concatenation qualified in resolving the mentioned issues by moderate hyper-parameter configurations. Our architecture devoid of stacking multiple layers achieved modest accuracy of 89.67% by DECR-Bi-GRU-CNN (WAP) on IMDB as compared to random initialization 81.11% on SST.

Flood Detection from Satellite Images Based on Deep Convolutional Neural Network and Layered Recurrent Neural Network

Satellite images are important for developing and protected environmental resources that can be used for flood detection. The satellite image of before-flooding and after-flooding to be segmented and feature with integration of deeply LRNN and CNN networks for giving high accuracy. It is also important for learning LRNN and CNN is able to find the feature of flooding regions sufficiently and, it will influence the effectiveness of flood relief. The CNNs and LRNNs consists of two set are training set and testing set. The before flooding and after flooding of satellite images to be extract and segment formed by testing and training phase of data patches. All patches are trained by LRNN where changes occur or any misdetection of flooded region to extract accurately without delay. This proposed method obtain accuracy of system is 99% of flood region detections.

One stage versus two stages deep learning approaches for the extraction of drug-drug interactions from texts

Drug-drug interactions (DDI) are a cause of adverse drug reactions. They occur when a drug has an impact on the effect of another drug. There is not a complete, up to date database where health care professionals can consult the interactions of any drug because most of the knowledge on DDI is hidden in unstructured text. In last years, deep learning has been succesfully applied to the extraction of DDI from texts, which requires the detection and later classification of DDI. Most of the deep learning systems for DDI extraction developed so far have addressed the detection and classification in one single step. In this study, we compare the performance of one-stage and two-stage architectures for DDI extraction. Our architectures are based on a bidirectional recurrent neural network layer composed of Gated Recurrent Units. The two-stage system obtained a 67.45 % micro-average F1 score on the test set.

Vision-Based Fatigue Driving Recognition Method Integrating Heart Rate and Facial Features

Driving fatigue can be detected by measuring drivers’ heart rate with a wearable device or extracting their facial features with an RGB camera. However, a wearable device causes inconvenience and discomfort to the driver, and an RGB camera’s detection accuracy may be affected by light, glasses, and head orientation. Furthermore, most existing methods ignored the temporal information of fatigue features and the relationship between the features, lowering recognition accuracy. Additionally, some existing fatigue detection methods focused on dealing with fatigue features with a temporal slice, ignoring temporal variations in the features. To address these problems, a single RGB-D camera is first used to extract three fatigue features: heart rate, eye openness level, and mouth openness level. More importantly, this paper proposes a novel multimodal fusion recurrent neural network (MFRNN), integrating the three features to improve the accuracy of driver fatigue detection. Specifically, a recurrent neural network (RNN) layer is applied in the MFRNN to obtain the temporal information of the features. Since the heart rate feature is a physiological signal extracted indirectly, it contains more noise and is fuzzier than the other features. To deal with the fuzziness and noise, we combine fuzzy reasoning with RNN to extract the temporal information of the heart rate. To identify the relationship between the features, we develop a new relationship layer containing a two-level RNN, for which the input is the temporal information of the features. Both the simulation and field experiment results show that the proposed method provides better performance than similar methods.

Recurrent attention network for false positive reduction in the detection of pulmonary nodules in thoracic CT scans.

Multiview two-dimensional (2D) convolutional neural networks (CNNs) and three-dimensional (3D) CNNs have been successfully used for analyzing volumetric data in many state-of-the-art medical imaging applications. We propose an alternative modular framework that analyzes volumetric data with an approach that is analogous to radiologists' interpretation, and apply the framework to reduce false positives that are generated in computer-aided detection (CADe) systems for pulmonary nodules in thoracic computed tomography (CT) scans. In our approach, a deep network consisting of 2D CNNs first processes slices individually. The features extracted in this stage are then passed to a recurrent neural network (RNN), thereby modeling consecutive slices as a sequence of temporal data and capturing the contextual information across all three dimensions in the volume of interest. Outputs of the RNN layer are weighed before the final fully connected layer, enabling the network to scale the importance of different slices within a volume of interest in an end-to-end training framework. We validated the proposed architecture on the false positive reduction track of the lung nodule analysis (LUNA) challenge for pulmonary nodule detection in chest CT scans, and obtained competitive results compared to 3D CNNs. Our results show that the proposed approach can encode the 3D information in volumetric data effectively by achieving a sensitivity >0.8 with just 1/8 false positives per scan. Our experimental results demonstrate the effectiveness of temporal analysis of volumetric images for the application of false positive reduction in chest CT scans and show that state-of-the-art 2D architectures from the literature can be directly applied to analyzing volumetric medical data. As newer and better 2D architectures are being developed at a much faster rate compared to 3D architectures, our approach makes it easy to obtain state-of-the-art performance on volumetric data using new 2D architectures.

Dynamic Terminal Sliding-Mode Control for Single-Phase Active Power Filter Using New Feedback Recurrent Neural Network

In this article, an adaptive dynamic terminal sliding-mode controller using a double hidden layer recurrent neural network (DHL-RNN) structure for a single-phase active power filter (APF) is proposed to improve harmonic compensation performance. First, a method combining dynamic sliding mode and terminal sliding mode is proposed to solve the chattering phenomenon in traditional sliding-mode control. Then, since the nonlinear dynamics of APF is difficult to obtain accurately, the DHL-RNN is used to approximate the proposed dynamic terminal sliding-mode controller. Meanwhile, an integral robust switching term is added to eliminate the approximation error of the neural network. Simulation and experimental results proved that the proposed controller has better compensation performance and tracking effect compared with a simple terminal sliding-mode controller.

Data augmentation for self-paced motor imagery classification with C-LSTM

Objective. Brain–computer interfaces (BCI) are becoming important tools for assistive technology, particularly through the use of motor imagery (MI) for aiding task completion. However, most existing methods of MI classification have been applied in a trial-wise fashion, with window sizes of approximately 2 s or more. Application of this type of classifier could cause a delay when switching between MI events. Approach. In this study, state-of-the-art classification methods for motor imagery are assessed offline with considerations for real-time and self-paced control, and a convolutional long-short term memory (C-LSTM) network based on filter bank common spatial patterns (FBCSP) is proposed. In addition, the effects of several methods of data augmentation on different classifiers are explored. Main results. The results of this study show that the proposed network achieves adequate results in distinguishing between different control classes, but both considered deep learning models are still less reliable than a Riemannian MDM classifier. In addition, controlled skewing of the data and the explored data augmentation methods improved the average overall accuracy of the classifiers by 14.0% and 5.3%, respectively. Significance. This manuscript is among the first to attempt combining convolutional and recurrent neural network layers for the purpose of MI classification, and is also one of the first to provide an in-depth comparison of various data augmentation methods for MI classification. In addition, all of these methods are applied on smaller windows of data and with consideration to ambient data, which provides a more realistic test bed for real-time and self-paced control.

Variational neural decoder for abstractive text summarization

In the conventional sequence-to-sequence (seq2seq) model for abstractive summarization, the internal transformation structure of recurrent neural networks (RNNs) is completely determined. Therefore, the learned semantic information is far from enough to represent all semantic details and context dependencies, resulting in a redundant summary and poor consistency. In this paper, we propose a variational neural decoder text summarization model (VND). The model introduces a series of implicit variables by combining variational RNN and variational autoencoder, which is used to capture complex semantic representation at each step of decoding. It includes a standard RNN layer and a variational RNN layer [5]. These two network layers respectively generate a deterministic hidden state and a random hidden state. We use these two RNN layers to establish the dependence between implicit variables between adjacent time steps. In this way, the model structure can better capture the complex semantics and the strong dependence between the adjacent time steps when outputting the summary, thereby improving the performance of generating the summary. The experimental results show that, on the text summary LCSTS and English Gigaword dataset, our model has a significant improvement over the baseline model.

Crop Yield Prediction Using Deep Reinforcement Learning Model for Sustainable Agrarian Applications

Predicting crop yield based on the environmental, soil, water and crop parameters has been a potential research topic. Deep-learning-based models are broadly used to extract significant crop features for prediction. Though these methods could resolve the yield prediction problem there exist the following inadequacies: Unable to create a direct non-linear or linear mapping between the raw data and crop yield values; and the performance of those models highly relies on the quality of the extracted features. Deep reinforcement learning provides direction and motivation for the aforementioned shortcomings. Combining the intelligence of reinforcement learning and deep learning, deep reinforcement learning builds a complete crop yield prediction framework that can map the raw data to the crop prediction values. The proposed work constructs a Deep Recurrent Q-Network model which is a Recurrent Neural Network deep learning algorithm over the Q-Learning reinforcement learning algorithm to forecast the crop yield. The sequentially stacked layers of Recurrent Neural network is fed by the data parameters. The Q- learning network constructs a crop yield prediction environment based on the input parameters. A linear layer maps the Recurrent Neural Network output values to the Q-values. The reinforcement learning agent incorporates a combination of parametric features with the threshold that assist in predicting crop yield. Finally, the agent receives an aggregate score for the actions performed by minimizing the error and maximizing the forecast accuracy. The proposed model efficiently predicts the crop yield outperforming existing models by preserving the original data distribution with an accuracy of 93.7%.