Layer Recurrent Neural Network Research Articles

Improved protein relative solvent accessibility prediction using deep multi-view feature learning framework

The accurate prediction of the relative solvent accessibility of a protein is critical to understanding its 3D structure and biological function. In this study, a novel deep multi-view feature learning (DMVFL) framework that integrates three different neural network units, i.e., bidirectional long short-term memory recurrent neural network, squeeze-and-excitation, and fully-connected hidden layer, with four sequence-based single-view features, i.e., position-specific scoring matrix, position-specific frequency matrix, predicted secondary structure, and roughly predicted three-state relative solvent accessibility probability, is developed to accurately predict relative solvent accessibility information of protein. On the basis of this newly developed framework, one new protein relative solvent accessibility predictor was proposed and called DMVFL-RSA, which employs a customized multiple feedback mechanism that helps to extract discriminative information embedded in the four single-view features. In benchmark tests on TEST524 and CASP14-derived (CASP14set) datasets, DMVFL-RSA outperforms other existing state-of-the-art protein relative solvent accessibility predictors when predicting two-state (exposure threshold of 25%), three-state (exposure thresholds of 9% and 36%), and four-state (exposure thresholds of 4%, 25%, and 50%) discrete values. For real-valued prediction on TEST524 and CASP14set, DMVFL-RSA has also gained high Pearson correlation coefficient values, indicating a positive correlation between the predicted and native relative solvent accessibility. Detailed analyses show that the major advantages of DMVFL-RSA lie in the high efficiency of the DMVFL framework, the applied multiple feedback mechanism, and the strong sensitivity of the sequence-based features. The web server of DMVFL-RSA is freely available at https://jun-csbio.github.io/DMVFL-RSA/for academic use. The standalone package of DMVFL-RSA is downloadable at https://github.com/XueQiangFan/DMVFL-RSA.

Compressing RNNs to Kilobyte Budget for IoT Devices Using Kronecker Products

Micro-controllers (MCUs) make up most of the processors in the world with widespread applicability from automobile to medical devices. The Internet of Things promises to enable these resource-constrained MCUs with machine learning algorithms to provide always-on intelligence. Many Internet of Things applications consume time-series data that are naturally suitable for recurrent neural networks (RNNs) like LSTMs and GRUs. However, RNNs can be large and difficult to deploy on these devices, as they have few kilobytes of memory. As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This article introduces a method to compress RNNs for resource-constrained environments using the Kronecker product (KP). KPs can compress RNN layers by 16× to 38× with minimal accuracy loss. By quantizing the resulting models to 8 bits, we further push the compression factor to 50×. We compare KP with other state-of-the-art compression techniques across seven benchmarks spanning five different applications and show that KP can beat the task accuracy achieved by other techniques by a large margin while simultaneously improving the inference runtime. Sometimes the KP compression mechanism can introduce an accuracy loss. We develop a hybrid KP approach to mitigate this. Our hybrid KP algorithm provides fine-grained control over the compression ratio, enabling us to regain accuracy lost during compression by adding a small number of model parameters.

Applying Convolutional-GRU for Term Deposit Likelihood Prediction

Banks are normally offered two kinds of deposit accounts. It consists of deposits like current/saving account and term deposits like fixed or recurring deposits.For enhancing the maximized profit from bank as well as customer perspective, term deposit can accelerate uplifting of finance fields. This paper focuses on likelihood of term deposit subscription taken by the customers. Bank campaign efforts and customer detail analysis caninfluence term deposit subscription chances. An automated system is approached in this paper that works towards prediction of term deposit investment possibilities in advance. This paper proposes deep learning based hybrid model that stacks Convolutional layers and Recurrent Neural Network (RNN) layers as predictive model. For RNN, Gated Recurrent Unit (GRU) is employed. The proposed predictive model is later compared with other benchmark classifiers such as k-Nearest Neighbor (k-NN), Decision tree classifier (DT), and Multi-layer perceptron classifier (MLP). Experimental study concludesthat proposed model attainsan accuracy of 89.59% and MSE of 0.1041 which outperform wellother baseline models.

Studying the Effect of Metre Perception on Rhythm and Melody Modelling with LSTMs

In this paper we take a connectionist machine learning approach to the problem of metre perception and melody learning in musical signals. We present a two-layered network consisting of a nonlinear oscillator network and a recurrent neural network. The oscillator network acts as an entrained resonant filter to the musical signal. It `perceives' metre by resonating nonlinearly to the inherent periodicities within the signal, creating a hierarchy of strong and weak periods. The neural network learns the long-term temporal structures present in this signal. We show that this network outperforms our previous approach of a single layer recurrent neural network in a melody and rhythm prediction task. We hypothesise that our system is enabled to make use of the relatively long temporal resonance in the oscillator network output, and therefore model more coherent long-term structures. A system such as this could be used in a multitude of analytic and generative scenarios, including live performance applications.

Towards multilingual end‐to‐end speech recognition for air traffic control

In this work, an end-to-end framework is proposed to achieve multilingual automatic speech recognition (ASR) in air traffic control (ATC) systems. Considering the standard ATC procedure, a recurrent neural network (RNN) based framework is selected to mine the temporal dependencies among speech frames. Facing the distributed feature space caused by the radio transmission, a hybrid feature embedding block is designed to extract high-level representations, in which multiple convolutional neural networks are designed to accommodate different frequency and temporal resolutions. The residual mechanism is performed on the RNN layers to improve the trainability and the convergence. To integrate the multilingual ASR into a single model and relieve the class imbalance, a special vocabulary is designed to unify the pronunciation of the vocabulary in Chinese and English, i.e., pronunciation-oriented vocabulary. The proposed model is optimized by the connectionist temporal classification loss and is validated on a real-world speech corpus (ATCSpeech). A character error rate of 4.4% and 5.9% is achieved for Chinese and English speech, respectively, which outperforms other popular approaches. Most importantly, the proposed approach achieves the multilingual ASR task in an end-to-end manner with considerable high performance.

A probabilistic Bayesian recurrent neural network for remaining useful life prognostics considering epistemic and aleatory uncertainties

Deep learning-based approach has emerged as a promising solution to handle big machinery data from multi-sensor suites in complex physical assets and predict their remaining useful life (RUL). However, most recent deep learning-based approaches deliver a single-point estimate of RUL as these models represent the weights of a neural network as a deterministic value and hence cannot convey uncertainty in the RUL prediction. This practice usually provides overly confident predictions that might cause severe consequences in safety-critical industries. To address this issue, this paper proposes a probabilistic Bayesian recurrent neural network (RNN) for RUL prognostics considering epistemic and aleatory uncertainties. The epistemic uncertainty is handled by Bayesian RNN layers as extensions from the Frequentist RNN layers using the Flipout method. The aleatory uncertainty is covered by a probabilistic output that follows a Gaussian distribution parameterized by the two neurons in the output layer. The network is trained using Bayes by backprop with the Flipout method. The proposed model is demonstrated by the open-access Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset of turbofan engines and a comparative study of the Frequentist RNN counterparts, the Monte Carlo Dropout-based RNN, and the state-of-the-art models for C-MAPSS datasets. The results demonstrate the promising performance and robustness of the proposed model in RUL prognostics.

RNN based Control Algorithm for Power Quality Improvement in PV Integrated Distribution System

In this paper, a single layer Recurrent Neural Network (RNN) algorithm has been proposed for the control of single-stage three-phase grid integrated solar photovoltaic (PV) system. Voltage Source converter (VSC) has been controlled for overcoming power quality problems in distribution systems viz. harmonic mitigation, active and reactive power compensation, load balancing etc. in three-phase grid connected PV systems. The PV array has been designed such that it delivers maximum power to the grid via VSC. The proposed algorithm has been implemented in MATLAB/Simulink and tested on hardware. Simulation and experimental results have been shown for the verification of the proposed algorithm under steady-state and dynamic conditions. These results conform to standard IEEE-1547 and the harmonics in grid currents are reduced to < 5% for the grid-connected PV system.

End-to-end speech emotion recognition using a novel context-stacking dilated convolution neural network

Amongst the various characteristics of a speech signal, the expression of emotion is one of the characteristics that exhibits the slowest temporal dynamics. Hence, a performant speech emotion recognition (SER) system requires a predictive model that is capable of learning sufficiently long temporal dependencies in the analysed speech signal. Therefore, in this work, we propose a novel end-to-end neural network architecture based on the concept of dilated causal convolution with context stacking. Firstly, the proposed model consists only of parallelisable layers and is hence suitable for parallel processing, while avoiding the inherent lack of parallelisability occurring with recurrent neural network (RNN) layers. Secondly, the design of a dedicated dilated causal convolution block allows the model to have a receptive field as large as the input sequence length, while maintaining a reasonably low computational cost. Thirdly, by introducing a context stacking structure, the proposed model is capable of exploiting long-term temporal dependencies hence providing an alternative to the use of RNN layers. We evaluate the proposed model in SER regression and classification tasks and provide a comparison with a state-of-the-art end-to-end SER model. Experimental results indicate that the proposed model requires only 1/3 of the number of model parameters used in the state-of-the-art model, while also significantly improving SER performance. Further experiments are reported to understand the impact of using various types of input representations (i.e. raw audio samples vs log mel-spectrograms) and to illustrate the benefits of an end-to-end approach over the use of hand-crafted audio features. Moreover, we show that the proposed model can efficiently learn intermediate embeddings preserving speech emotion information.

Stacked recurrent neural network for botnet detection in smart homes

Internet of Things (IoT) devices in Smart Home Network (SHN) are highly vulnerable to complex botnet attacks. In this paper, we investigate the effectiveness of Recurrent Neural Network (RNN) to correctly classify network traffic samples in the minority classes of highly imbalanced network traffic data. Multiple layers of RNN are stacked to learn the hierarchical representations of highly imbalanced network traffic data with different levels of abstraction. We evaluate the performance of Stacked RNN (SRNN) model with Bot-IoT dataset. Results show that SRNN outperformed RNN in all classification scenarios. Specifically, SRNN model learned the discriminating features of highly imbalanced network traffic samples in the training set with better representations than RNN model. Also, SRNN model is more robust and it demonstrated better capability to effectively handle over-fitting problem than RNN model. Furthermore, SRNN model achieved better generalization ability in detecting network traffic samples of the minority classes.

Quality Enhancement of Compressed Vibrotactile Signals Using Recurrent Neural Networks and Residual Learning.

We present a neural network-based compression artifact removal technique for vibrotactile signals. The proposed decoder-side quality enhancement approach is based on recurrent neural networks (RNNs) and the principle of residual learning. We use a total of 8 nonlinear RNN layers trained to first estimate the difference between the original and the compressed signal. The estimated difference signal is then added to the compressed signal, followed by further linear processing steps to construct the enhanced signal. With our approach, we are able to enhance signals at almost all compression ratios by up to $1.25\ \mathrm {dB}$. For the signals in our data set, rougly 86% are enhanced in their quality. Through an ablation study, we show that every block of our network is functioning as intended and contributes to the compression artifact removal. Additionally, we show that the chosen network parameters maximize performance.

Forecasting of COVID-19 using deep layer Recurrent Neural Networks (RNNs) with Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) cells.

In December 2019, first case of the COVID-19 was reported in Wuhan, Hubei province in China. Soon world health organization has declared contagious coronavirus disease (a.k.a. COVID-19) as a global pandemic in the month of March 2020. Over the span of eleven months, it has rapidly spread out all over the world with total confirmed cases of ~ 41.39 M and causing a total fatality of ~1.13 M. At present, the entire mankind is facing serious threat and it is believed that COVID-19 may have been around for quite some time. Therefore, it has become imperative to forecast the global impact of COVID-19 in the near future. The present work proposes state-of-art deep learning Recurrent Neural Networks (RNN) models to predict the country-wise cumulative confirmed cases, cumulative recovered cases and the cumulative fatalities. The Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) cells along with Recurrent Neural Networks (RNN) were developed to predict the future trends of the COVID-19. We have used publicly available data from John Hopkins University's COVID-19 database. In this work, we emphasize the importance of various factors such as age, preventive measures, and healthcare facilities, population density, etc. that play vital role in rapid spread of COVID-19 pandemic. Therefore, our forecasted results are very helpful for countries to better prepare themselves to control the pandemic.

Fuzzy Multiple Hidden Layer Recurrent Neural Control of Nonlinear System Using Terminal Sliding-Mode Controller.

This study designs a fuzzy double hidden layer recurrent neural network (FDHLRNN) controller for a class of nonlinear systems using a terminal sliding-mode control (TSMC). The proposed FDHLRNN is a fully regulated network, which can be simply considered as a combination of a fuzzy neural network (FNN) and a radial basis function neural network (RBF NN) to improve the accuracy of a nonlinear approximation, so it has the advantages of these two neural networks. The main advantage of the proposed new FDHLRNN is that the output values of the FNN and DHLRNN are considered at the same time, and the outer layer feedback is added to increase the dynamic approximation ability. FDHLRNN was designed to approximate the nonlinear sliding-mode equivalent control term to reduce the switching gain. To ensure the best approximation capability and control performance, the proposed FDHLRNN using TSMC is applied for the second-order nonlinear model. Two simulation examples are implemented to verify that the proposed FDHLRNN has faster convergence speed and the FDHLRNN with TSMC has good dynamic property and robustness, and a hardware experimental study with an active power filter proves the feasibility of the method.

Enhanced Harris Hawks optimization as a feature selection for the prediction of student performance

Predicting student performance for educational organizations such as universities, community colleges, schools, and training centers will enhance the overall results of these organizations. Big data can be extracted from the internal systems of these organizations, such as exam records, statistics about virtual courses, and e-learning systems. Finding meaningful knowledge from extracted data is a challenging task. In this paper, we proposed a modified version of Harris Hawks Optimization (HHO) algorithm by controlling the population diversity to overcome the early convergence problem and prevent trapping in a local optimum. The proposed approach is employed as a feature selection algorithm to discover the most valuable features for student performance prediction problem. A dynamic controller that controls the population diversity by observing the performance of HHO using the k-nearest neighbors (kNN) algorithm as a clustering approach. Once all solutions belong to one cluster, an injection process is employed to redistribute the solutions over the search space. A set of machine learning classifiers such as kNN, Layered recurrent neural network (LRNN), Naive Bayes, and Artificial Neural Network are used to evaluate the overall prediction system. A real dataset obtained from UCI machine learning repository is adopted in this paper. The obtained results show the importance of predicting students’ performance at an earlier stage to avoid students’ failure and improve the overall performance of the educational organization. Moreover, the reported results show that the combination between the enhanced HHO and LRNN can outperform other classifiers with accuracy equal to $$92\%$$ , since LRNN is a deep learning algorithm that is able to learn from previous and current input values.

CRF-EfficientUNet: An Improved UNet Framework for Polyp Segmentation in Colonoscopy Images With Combined Asymmetric Loss Function and CRF-RNN Layer

Colonoscopy is considered the gold-standard investigation for colorectal cancer screening. However, the polyps miss rate in clinical practice is relatively high due to different factors. This presents an opportunity to use AI models to automatically detect and segment polyps, supporting clinicians to reduce the number of polyps missed. Inspired by the success of UNets, a popular strategy for solving medical image segmentation tasks, this article proposes a novel framework for polyp segmentation called CRF-EfficientUNet, which enhances UNet using the EfficientNet encoder, a combined asymmetric loss function, and Conditional Random Field as a Recurrent Neural Network (CRF-RNN) layer on top. A novel loss function that combines pixel-wise cross-entropy loss and asymmetric similarity loss to solve the unbalanced imaging data problem is proposed. Training the proposed network with this loss function can achieve a considerably higher Dice score and better polyp segmentation prediction. In addition, we add the CRF-RNN layer to the proposed framework to improve the quality of semantic segmentation. Experimental results on popular benchmark datasets show that CRF-EfficientUNet achieves state-of-the-art accuracy compared to existing methods. The results of the experiments, which are performed on the CVC-ClinicDB dataset for training and testing, are 95.55% Dice and 92.23% IoU. While the experimental results on cross-dataset using Kvasir-SEG as the training set, CVC-ColonDB as the test set are 85.59% Dice and 76.19% IoU. These results indicate that the proposed method has high generalization capability and learning ability, and it can be a compelling choice for practical applications with considerable data variations.

Fuzzy Multiple Hidden Layer Neural Sliding Mode Control of Active Power Filter With Multiple Feedback Loop

A fuzzy multiple hidden layer neural sliding mode control with multiple feedback loop (FMHLNSMCMFL) is proposed for a single-phase active power filter (APF), where a sliding mode controller is designed to make the current tracking error converge to zero and a new neural network with multiple feedback loops is introduced to approximate unknown dynamics. At the same time, the fuzzy neural network can eliminate chattering, improve the control accuracy and reduce the current distortion rate of APF. Moreover, the proposed double feedback fuzzy double hidden layer recurrent neural network is the weighted sum of fuzzy network and double hidden layer network and has strong global learning ability. The adaptive parameters obtained by Lyapunov function can ensure the asymptotic stability of the system. Simulation and hardware experiments verify the introduced FMHLNSMCMFL scheme is a viable control solution for the APF.

LocalGLMnet: A Deep Learning Architecture for Actuaries

The purpose of this tutorial is to discuss the LocalGLMnet architecture which is tailored to the needs of actuaries. The LocalGLMnet is a flexible network architecture for tabular data that allows for variable selection, the study of interactions, gives nice interpretations and allows to rank variable importance. We explore a LocalGLMnet on accident insurance claims data for which we also have short claim descriptions. In a second step we try to understand the predictive power of these claim descriptions by adding a recurrent neural network layer to process the claim texts into tabular data.

Attention-based Convolutional Neural Network for MRI Gibbs-ringing Artifact Suppression

Gibbs-ringing artifact is a common artifact in MRI image processing. As MRI raw data is taken in a frequency domain, 2D in- verse discrete Fourier transform is applied to visualize data. Inability to take inverse Fourier transform of full spectrum (full k-space) leads to the insufficient sampling of the high frequency data and results in a well-known Gibbs phenomenon. It is worth to notice that truncation of high frequency information generates a significant blur, thus some techniques from other image restoration problems (for example, image deblur task) can be successfully used. We propose attention-based convolutional neural network for Gibbs-ringing reduction which is the extension of recently proposed GAS-CNN (Gibbs-ringing Artifact Suppression Convolutional Neural Network). Proposed method includes simplified non-linear mapping, amended by LRNN (Layer Recurrent Neural Network) refinement block with feature attention module, controlling the correlation between input and output tensors of the refinement unit. The research shows that the proposed post-processing refinement construction considerably simplifies the non-linear mapping.

Forecasting Spatially-Distributed Urban Traffic Volumes via Multi-Target LSTM-Based Neural Network Regressor

Monitoring the distribution of vehicles across the city is of great importance for urban traffic control. In particular, information on the number of vehicles entering and leaving a city, or moving between urban areas, gives a valuable estimate on potential bottlenecks and congestions. The possibility of predicting such flows in advance is even more beneficial, allowing for timely traffic management strategies and targeted congestion warnings. Our work is inserted in the context of short-term forecasting, aiming to predict rapid changes and sudden variations in the traffic volume, beyond the general trend. Moreover, it concurrently targets multiple locations in the city, providing an instant prediction outcome comprising the future distribution of vehicles across several urban locations. Specifically, we propose a multi-target deep learning regressor for simultaneous predictions of traffic volumes, in multiple entry and exit points among city neighborhoods. The experiment focuses on an hourly forecasting of the amount of vehicles accessing and moving between New York City neighborhoods through the Metropolitan Transportation Authority (MTA) bridges and tunnels. By leveraging a single training process for all location points, and an instant one-step volume inference for every location at each time update, our sequential modeling approach is able to grasp rapid variations in the time series and process the collective information of all entry and exit points, whose distinct predicted values are outputted at once. The multi-target model, based on long short-term memory (LSTM) recurrent neural network layers, was tested on a real-world dataset, achieving an average prediction error of 7% and demonstrating its feasibility for short-term spatially-distributed urban traffic forecasting.

Developing an Individual Glucose Prediction Model Using Recurrent Neural Network.

In this study, we propose a personalized glucose prediction model using deep learning for hospitalized patients who experience Type-2 diabetes. We aim for our model to assist the medical personnel who check the blood glucose and control the amount of insulin doses. Herein, we employed a deep learning algorithm, especially a recurrent neural network (RNN), that consists of a sequence processing layer and a classification layer for the glucose prediction. We tested a simple RNN, gated recurrent unit (GRU), and long-short term memory (LSTM) and varied the architectures to determine the one with the best performance. For that, we collected data for a week using a continuous glucose monitoring device. Type-2 inpatients are usually experiencing bad health conditions and have a high variability of glucose level. However, there are few studies on the Type-2 glucose prediction model while many studies performed on Type-1 glucose prediction. This work has a contribution in that the proposed model exhibits a comparative performance to previous works on Type-1 patients. For 20 in-hospital patients, we achieved an average root mean squared error (RMSE) of 21.5 and an Mean absolute percentage error (MAPE) of 11.1%. The GRU with a single RNN layer and two dense layers was found to be sufficient to predict the glucose level. Moreover, to build a personalized model, at most, 50% of data are required for training.

Prognostic of RUL based on Echo State Network Optimized by Artificial Bee Colony

Prognostic is an engineering technique used to predict the future health state or behavior of an equipment or system. In this work, a data-driven hybrid approach for prognostic is presented. The approach based on Echo State Network (ESN) and Artificial Bee Colony (ABC) algorithm is used to predict machine’s Remaining Useful Life (RUL). ESN is a new paradigm that establishes a large space dynamic reservoir to replace the hidden layer of Recurrent Neural Network (RNN). Through the application of ESN is possible to overcome the shortcomings of complicated computing and difficulties in determining the network topology of traditional RNN. This approach describes the ABC algorithm as a tool to set the ESN with optimal parameters. Historical data collected from sensors are used to train and test the proposed hybrid approach in order to estimate the RUL. To evaluate the proposed approach, a case study was carried out using turbofan engine signals show that the proposed method can achieve a good collected from physical sensors (temperature, pressure, speed, fuel flow, etc.). The experimental results using the engine data from NASA Ames Prognostics Data Repository RUL estimation precision. The performance of this model was compared using prognostic metrics with the approaches that use the same dataset. Therefore, the ESN-ABC approach is very promising in the field of prognostics of the RUL.