Output Of Recurrent Neural Network Research Articles

Use of Parallel Explanatory Models to Enhance Transparency of Neural Network Configurations for Cell Degradation Detection.

In a previous paper, we have shown that a recurrent neural network (RNN) can be used to detect cellular network radio signal degradations accurately. We unexpectedly found, though, that accuracy gains diminished as we added layers to the RNN. To investigate this, in this article, we build a parallel model to illuminate and understand the internal operation of neural networks (NNs), such as the RNN, which store their internal state to process sequential inputs. This model is widely applicable in that it can be used with any input domain where the inputs can be represented by a Gaussian mixture. By looking at RNN processing from a probability density function (pdf) perspective, we are able to show how each layer of the RNN transforms the input distributions to increase detection accuracy. At the same time we also discover a side effect acting to limit the improvement in accuracy. To demonstrate the fidelity of the model, we validate it against each stage of RNN processing and output predictions. As a result, we have been able to explain the reasons for RNN performance limits with useful insights for future designs for RNNs and similar types of NN.

Optimization of RNN-LSTM Model Using NSGA-II Algorithm for IOT-based Fire Detection Framework

The occurrence of fire leads to unparalleled loss of resources as well as human life and hence, fire detection systems must be trustworthy and less erroneous. Real-time assessment of fire conditions through predictive learning models could lead to easier decision-making and timely rescue operations. Reported works are often restricted to the use of singular rule-based algorithms which can hardly offer a comprehensive solution by adapting to changing dynamics of fire conditions due to their static features, mostly leading to inaccurate classification. In this research, an efficient fire prediction framework has been proposed by efficiently combining the outputs of recurrent neural networks which bear the advantages of short-term predictions with long short-term memory that takes care of long-term predictions. Weights to combine multi-objective optimization functions to minimize mean absolute error and root mean square error have been designed using non-dominated sorting genetic algorithm II (NSGA-II) which offers a lower time complexity. The benchmark dataset from the NIST website has been chosen for analysis and performance validation of the proposed classifier on experimental data pertaining to different fire scenarios. A Pareto optimal front has been obtained from the proposed algorithm which represents the optimum solutions. The performance of the proposed model has been exhaustively evaluated through different factors such as accuracy, RMSE, MAE, F-Measure, binary classification rate, negative predictive value, recall and precision which justifies its contribution. The proposed model reduced RMSE by 14.95–19.88% compared to baseline machine learning models along with an enhanced accuracy of 95.05% and reduced false positive rate which is better compared to reported works along with improvement in F-Measure. Results show that the proposed NSGA-II-based RNN LSTM model accurately predicts the occurrence of fire events with reduced false alarms while maintaining a low computational overhead.

Combined Shallow and Deep Learning Models for Malware Detection in Wsn

Due to the major operating restrictions, ensuring security is the fundamental problem of Wireless Sensor Networks (WSNs). Because of their inadequate security mechanisms, WSNs are indeed a simple point for malware (worms, viruses, malicious code, etc.). According to the epidemic nature of worm propagation, it is critical to develop a worm defense mechanism in the network. This concept aims to establish novel malware detection in WSN that consists of several phases: “(i) Preprocessing, (ii) feature extraction, as well as (iii) detection”. At first, the input data is subjected for preprocessing phase. Then, the feature extraction takes place, in which principal component analysis (PCA), improved linear discriminant analysis (LDA), and autoencoder-based characteristics are retrieved. Moreover, the retrieved characteristics are subjected to the detection phase. The detection is performed employing combined shallow learning and DL. Further, the shallow learning includes decision tree (DT), logistic regression (LR), and Naive Bayes (NB); the deep learning (DL) includes deep neural network (DNN), convolutional neural network (CNN), and recurrent neural network (RNN). Here, the DT output is given to the DNN, LR output is subjected to CNN, and the NB output is given to the RNN, respectively. Eventually, the DNN, CNN, and RNN outputs are averaged to generate a successful outcome. The combination can be thought of as an Ensemble classifier. The weight of the RNN is optimally tuned through the Self Improved Shark Smell Optimization with Opposition Learning (SISSOOL) model to improve detection precision and accuracy. Lastly, the outcomes of the suggested approach are computed in terms of different measures.

A Guided Tutorial on Modelling Human Event-Related Potentials with Recurrent Neural Networks

In cognitive neuroscience research, computational models of event-related potentials (ERP) can provide a means of developing explanatory hypotheses for the observed waveforms. However, researchers trained in cognitive neurosciences may face technical challenges in implementing these models. This paper provides a tutorial on developing recurrent neural network (RNN) models of ERP waveforms in order to facilitate broader use of computational models in ERP research. To exemplify the RNN model usage, the P3 component evoked by target and non-target visual events, measured at channel Pz, is examined. Input representations of experimental events and corresponding ERP labels are used to optimize the RNN in a supervised learning paradigm. Linking one input representation with multiple ERP waveform labels, then optimizing the RNN to minimize mean-squared-error loss, causes the RNN output to approximate the grand-average ERP waveform. Behavior of the RNN can then be evaluated as a model of the computational principles underlying ERP generation. Aside from fitting such a model, the current tutorial will also demonstrate how to classify hidden units of the RNN by their temporal responses and characterize them using principal component analysis. Statistical hypothesis testing can also be applied to these data. This paper focuses on presenting the modelling approach and subsequent analysis of model outputs in a how-to format, using publicly available data and shared code. While relatively less emphasis is placed on specific interpretations of P3 response generation, the results initiate some interesting discussion points.

Cognitive workload classification: Towards generalization through innovative pipeline interface using HMM

Cognition is an important aspect to realize one’s potential and accelerate progress towards achieving a better quality of life. The cognitive workload is characterized by the occupancy of working memory during task performance. In spite of low spatial resolution, Electroencephalogram (EEG) continues to be a substantial tool in cognitive load research. The current methods use various time and frequency domain features for the assessment of cognitive load levels using EEG. The present work intends to model the temporal dynamics of the EEG signals by proposing a pipeline interface to classify cognitive workload under a continuous engagement/attention environment. The short segments of decomposed data are assumed to be stationary and are modeled using the autoregressive method. Later, fuzzy c-means maps the AR coefficients to low dimensional space. These clustered items are scaled further in the temporal domain by finding the interplay between the successive items through the most likely sequence of hidden states using the hidden Markov model (HMM). The successive segments group, having similar HMM states form variable-length frames. The handcrafted feature set from variable-length frames is classified using a deep recurrent neural network (RNN) structure. Finally, the maximum voting scheme on the predicted RNN output enhances the classification accuracy. The performance of this method is evaluated on a publicly available, and self-collected dataset. The findings reveal that handcrafted features supported by modeling of temporal dynamics outperform state-of-the-art methods. As a result, 97.8% accuracy is observed for multi-class classification with an optimum of four electrodes.

Subtraction Gates: Another Way to Learn Long-Term Dependencies in Recurrent Neural Networks.

Recurrent neural networks (RNNs) can remember temporal contextual information over various time steps. The well-known gradient vanishing/explosion problem restricts the ability of RNNs to learn long-term dependencies. The gate mechanism is a well-developed method for learning long-term dependencies in long short-term memory (LSTM) models and their variants. These models usually take the multiplication terms as gates to control the input and output of RNNs during forwarding computation and to ensure a constant error flow during training. In this article, we propose the use of subtraction terms as another type of gates to learn long-term dependencies. Specifically, the multiplication gates are replaced by subtraction gates, and the activations of RNNs input and output are directly controlled by subtracting the subtrahend terms. The error flows remain constant, as the linear identity connection is retained during training. The proposed subtraction gates have more flexible options of internal activation functions than the multiplication gates of LSTM. The experimental results using the proposed Subtraction RNN (SRNN) indicate comparable performances to LSTM and gated recurrent unit in the Embedded Reber Grammar, Penn Tree Bank, and Pixel-by-Pixel MNIST experiments. To achieve these results, the SRNN requires approximate three-quarters of the parameters used by LSTM. We also show that a hybrid model combining multiplication forget gates and subtraction gates could achieve good performance.

Physiological Signal Emotion Recognition Based on Temporal Convolutional Networks

Abstract Human emotion recognition has become an important research field. Because of its objectivity, physiological signals have become one of the most robust cues for emotion recognition. In recent years, deep learning methods have made great progress in the field of emotion recognition, especially the superiority of recurrent neural networks (RNN) in time series models, and more and more tasks are completed based on RNN. However, RNN has problems such as time-consuming and gradient disappearance and explosion, and the feature input and sentiment output of RNN are not aligned. To avoid these problems, this paper is based on the temporal convolutional networks (TCN) model and the connectionist temporal classification (CTC) algorithm to process the emotion recognition task. First, generate a spectrogram representation of the physiological signal in each channel; Second, use the TCN to learn long-term dynamic features, and use CTC to align the dynamic features and their sentiment labels, and then feed the learned deep features into the neural network to predict the sentiment of each channel; Finally, take the best result as the final emotion representation. Experimental results on the AMIGOS dataset show that the proposed method outperforms existing methods.

Recurrent Neural Networks (RNNs) with dimensionality reduction and break down in computational mechanics; application to multi-scale localization step

Artificial Neural Networks (NNWs) are appealing functions to substitute high dimensional and non-linear history-dependent problems in computational mechanics since they offer the possibility to drastically reduce the computational time. This feature has recently been exploited in the context of multi-scale simulations, in which the NNWs serve as surrogate model of micro-scale finite element resolutions. Nevertheless, in the literature, mainly the macro-stress–macro-strain response of the meso-scale boundary value problem was considered and the micro-structure information could not be recovered in a so-called localization step. In this work, we develop Recurrent Neural Networks (RNNs) as surrogates of the RVE response while being able to recover the evolution of the local micro-structure state variables for complex loading scenarios. The main difficulty is the high dimensionality of the RNNs output which consists in the internal state variable distribution in the micro-structure. We thus propose and compare several surrogate models based on a dimensionality reduction: (i) direct RNN modeling with implicit NNW dimensionality reduction, (ii) RNN with PCA dimensionality reduction, and (iii) RNN with PCA dimensionality reduction and dimensionality break down, i.e. the use of several RNNs instead of a single one. Besides, we optimize the sequential training strategy of the latter surrogate for GPU usage in order to speed up the process. Finally, through RNN modeling of the principal components coefficients, the connection between the physical state variables and the hidden variables of the RNN is revealed, and exploited in order to select the hyper-parameters of the RNN-based surrogate models in their design stage.

Financial Instrument Forecast with Artificial Intelligence

In ancient times, trade was carried out by barter. With the use of money and similar means, the concept of financial instruments emerged. Financial instruments are tools and documents used in the economy. Financial instruments can be foreign exchange rates, securities, crypto currency, index and funds. There are many methods used in financial instrument forecast. These methods include technical analysis methods, basic analysis methods, forecasts carried out using variables and formulas, time-series algorithms and artificial intelligence algorithms. Within the scope of this study, the importance of the use of artificial intelligence algorithms in the financial instrument forecast is studied. Since financial instruments are used as a means of investment and trade by all sections of the society, namely individuals, families, institutions, and states, it is highly important to know about their future. Financial instrument forecast can bring about profitability such as increased income welfare, more economical adjustment of maturities, creation of large finances, minimization of risks, spreading of ownership to the grassroots, and more balanced income distribution. Within the scope of this study, financial instrument forecast is carried out by applying a new methods of Long Short Term Memory (LSTM), Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), Autoregressive Integrated Moving Average (ARIMA) algorithms and Ensemble Classification Boosting Method. Financial instrument forecast is carried out by creating a network compromising LSTM and RNN algorithm, an LSTM layer, and an RNN output layer. With the ensemble classification boosting method, a new method that gives a more successful result compared to the other algorithm forecast results was applied. At the conclusion of the study, alternative algorithm forecast results were competed against each other and the algorithm that gave the most successful forecast was suggested. The success rate of the forecast results was increased by comparing the results with different time intervals and training data sets. Furthermore, a new method was developed using the ensemble classification boosting method, and this method yielded a more successful result than the most successful algorithm result.

Automated Software Vulnerability Detection Based on Hybrid Neural Network

Vulnerabilities threaten the security of information systems. It is crucial to detect and patch vulnerabilities before attacks happen. However, existing vulnerability detection methods suffer from long-term dependency, out of vocabulary, bias towards global features or local features, and coarse detection granularity. This paper proposes an automatic vulnerability detection framework in source code based on a hybrid neural network. First, the inputs are transformed into an intermediate representation with explicit structure information using lower level virtual machine intermediate representation (LLVM IR) and backward program slicing. After the transformation, the size of samples and the size of vocabulary are significantly reduced. A hybrid neural network model is then applied to extract high-level features of vulnerability, which learns features both from convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The former is applied to learn local vulnerability features, such as buffer size. Furthermore, the latter is utilized to learn global features, such as data dependency. The extracted features are made up of concatenated outputs of CNN and RNN. Experiments are performed to validate our vulnerability detection method. The results show that our proposed method achieves excellent results with F1-scores of 98.6% and accuracy of 99.0% on the SARD dataset. It outperforms state-of-the-art methods.

Intelligent hybrid technique for cascaded multilevel inverter based three phase grid-tie hybrid power system: a WPSNN technique

This dissertation introduces an intelligent hybrid technique for the design of modelling and controller of the three-phase hybrid network connection system based on CMI. The hybrid technique is the Recurrent Neural Network (RNN) assisted by Wolf Pack Search (WPS) and therefore it is known as WPSNN technique. Here, the three stage CMI control scheme is considered as two stages and these are analyzed with the help of the proposed intelligent controller. In first stage, the WPS is developed to demonstrating the PV total and wind voltages and the WPS algorithm of the second stage assisted to RNN is used to extracting the reference currents of network. In first stage, input of WPS algorithm is PV and wind reference voltages and the gain parameters is generated as the output for optimal tuning of PI controller. In the second phase, the RNN is trained with on line process; the output of RNN is given to the WPS algorithm. It provides the corresponding tuning parameters to achieve the network current. To regulate the dc link voltage and the extraction of network current is developed with established technique. CMI based on the established technique is synchronizes the firing work relationship and reduces the modulation load. By then, the established technique is evaluated in MATLAB/Simulink working stage and the execution is assessed to existing techniques.

Database security: combining neural networks and classification approach

In the paper we present a new approach based on application of neural networks to detect SQL attacks. SQL attacks are those attacks that take the advantage of using SQL statements to be performed. The problem of detection of this class of attacks is transformed to time series prediction problem. SQL queries are used as a source of events in a protected environment. To differentiate between normal SQL queries and those sent by an attacker, we divide SQL statements into tokens and pass them to our detection system, which predicts the next token, taking into account previously seen tokens. In the learning phase tokens are passed to a recurrent neural network (RNN) trained by backpropagation through time (BPTT) algorithm. Then, two coefficients of the rule are evaluated. The rule is used to interpret RNN output. In the testing phase RNN with the rule is examined against attacks and legal data to find out how evaluated rule affects efficiency of detecting attacks. All experiments were conducted on Jordan network. Experimental results show the relationship between the rule and a length of SQL queries.

Speech emotion recognition using fusion of three multi-task learning-based classifiers: HSF-DNN, MS-CNN and LLD-RNN

Speech emotion recognition plays an increasingly important role in emotional computing and is still a challenging task due to its complexity. In this study, we developed a framework integrating three distinctive classifiers: a deep neural network (DNN), a convolution neural network (CNN), and a recurrent neural network (RNN). The framework was used for categorical recognition of four discrete emotions (i.e., angry, happy, neutral and sad). Frame-level low-level descriptors (LLDs), segment-level mel-spectrograms (MS), and utterance-level outputs of high-level statistical functions (HSFs) on LLDs were passed to RNN, CNN, and DNN, separately. Three individual models of LLD-RNN, MS-CNN, and HSF-DNN were obtained. In the models of MS-CNN and LLD-RNN, the attention mechanism based weighted-pooling method was utilized to aggregate the CNN and RNN outputs. To effectively utilize the interdependencies between the two approaches of emotion description (discrete emotion categories and continuous emotion attributes), a multi-task learning strategy was implemented in these three models to acquire generalized features by simultaneously operating classification of discrete categories and regression of continuous attributes. Finally, a confidence-based fusion strategy was developed to integrate the power of different classifiers in recognizing different emotional states. Three experiments on emotion recognition based on the IEMOCAP corpus were conducted. Our experimental results show that the weighted pooling method based on attention mechanism endowed the neural networks with the capability to focus on emotionally salient parts. The generalized features learned in the multi-task learning helped the neural networks to achieve higher accuracies in the tasks of emotion classification. Furthermore, our proposed fusion system achieved weighted accuracy of 57.1% and unweighted accuracy of 58.3%, which were significantly higher than those of each individual classifier. The effectiveness of the proposed approach based on classifier fusion was thus validated.

Trading Saudi Stock Market Shares using Multivariate Recurrent Neural Network with a Long Short-term Memory Layer

This study tests the Saudi stock market weak form using the weak form of an efficient market hypothesis and proposes a recurrent neural network (RNN) to produce a trading signal. To predict the next-day trading signal of several shares in the Saudi stock market, we designed the RNN with a long short-term memory architecture. The network input comprises several time series features that contribute to the classification process. The proposed RNN output is fed to a trading agent that buys or sells shares based on the share current value, current available balance, and the current number of shares owned. To evaluate the proposed neural network, we used the historical oil price data of Brent crude oil in combination with other stock features (e.g., previous day (opening and closing price of the evaluated share). The results indicate that oil price variations affect the Saudi stock market. Furthermore, with 55% accuracy, the proposed RNN model produces the next-day trading signal. For the same period, the proposed RNN trading method achieves an investment gain of 23%, whereas the buy-and-hold method obtained 1.2%.

Fast Query-by-example Speech Search using Attention-based Deep Binary Embeddings

State-of-the-art query-by-example (QbE) speech search approaches usually use recurrent neural network (RNN) based acoustic word embeddings (AWEs) to represent variable-length speech segments with fixed-dimensional vectors, and thus simple cosine distances can be measured over the embedded vectors of both the spoken query and the search content. In this paper, we aim to improve search accuracy and speed for the AWE-based QbE approach in low-resource scenario. First, multi-head self-attentive mechanism is introduced for learning a sequence of attention weights for all time steps of RNN outputs while attending to different positions of a speech segment. Second, as the real-valued AWEs suffer from substantial computation in similarity measure, a hashing layer is adopted for learning deep binary embeddings, and thus binary pattern matching can be directly used for fast QbE speech search. The proposed approach of self-attentive deep hashing network is effectively trained with three specifically-designed objectives: a penalization term, a triplet loss, and a quantization loss. Experiments show that our approach improves the relative search speed by 8 times and mean average precision (MAP) by 18.9%, as compared with the previous best real-valued embedding approach.

Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection

This paper proposes a segregated temporal assembly recurrent (STAR) network for weakly-supervised multiple action detection. The model learns from untrimmed videos with only supervision of video-level labels and makes prediction of intervals of multiple actions. Specifically, we first assemble video clips according to class labels by an attention mechanism that learns class-variable attention weights and thus helps the noise relieving from background or other actions. Secondly, we build temporal relationship between actions by feeding the assembled features into an enhanced recurrent neural network. Finally, we transform the output of recurrent neural network into the corresponding action distribution. In order to generate more precise temporal proposals, we design a score term called segregated temporal gradient-weighted class activation mapping (ST-GradCAM) fused with attention weights. Experiments on THUMOS’14 and ActivityNet1.3 datasets show that our approach outperforms the state-of-theart weakly-supervised method, and performs at par with the fully-supervised counterparts.

Local online learning in recurrent networks with random feedback.

Recurrent neural networks (RNNs) enable the production and processing of time-dependent signals such as those involved in movement or working memory. Classic gradient-based algorithms for training RNNs have been available for decades, but are inconsistent with biological features of the brain, such as causality and locality. We derive an approximation to gradient-based learning that comports with these constraints by requiring synaptic weight updates to depend only on local information about pre- and postsynaptic activities, in addition to a random feedback projection of the RNN output error. In addition to providing mathematical arguments for the effectiveness of the new learning rule, we show through simulations that it can be used to train an RNN to perform a variety of tasks. Finally, to overcome the difficulty of training over very large numbers of timesteps, we propose an augmented circuit architecture that allows the RNN to concatenate short-duration patterns into longer sequences.

Neural Session-Aware Recommendation

Recommender systems help users find items they are likely to interact within the near future, such as products to buy in e-commerce or songs to play in music websites. The Traditional recommendation methods make predictions based on long-term user profiles, i.e., the items a user interacted with in the past while ignoring the time and order of the interactions. Recent findings, however, suggest that users may exhibit interest to items in a certain order depending on situations and more recent items in a sequence have a larger impact on the subsequent choices. Moreover, in many practical applications, user-item interactions are organized into short sessions, where each session reflects the user's short-term interest in addition to long-term preferences. Leveraging both long-term user profiles and short-term sequential patterns from sessions can lead to more accurate models known as the session-aware recommendation methods. In this paper, we explore various strategies to integrate user long-term preferences with session patterns encoded by recurrent neural networks (RNNs). The strategies include integrating user embeddings with input and output of session RNNs, integrating with fixed or adaptive contributions of the user and session components by using a specially designed gating mechanism. We conducted an empirical evaluation of three publicly available datasets. The results indicate that combining user long-term profiles with the output of session RNNs yields improved predictions and the proposed adaptive integration model outperforms the state-of-the-art sequential and session-aware recommendation methods.

A Hardware-Oriented and Memory-Efficient Method for CTC Decoding

The Connectionist Temporal Classification (CTC) has achieved great success in sequence to sequence analysis tasks such as automatic speech recognition (ASR) and scene text recognition (STR). These applications can use the CTC objective function to train the recurrent neural networks (RNNs), and decode the outputs of RNNs during inference. While hardware architectures for RNNs have been studied, hardware-based CTCdecoders are desired for high-speed CTC-based inference systems. This paper, for the first time, provides a low-complexity and memory-efficient approach to build a CTC-decoder based on the beam search decoding. Firstly, we improve the beam search decoding algorithm to save the storage space. Secondly, we compress a dictionary (reduced from 26.02MB to 1.12MB) and use it as the language model. Meanwhile searching this dictionary is trivial. Finally, a fixed-point CTC-decoder for an English ASR and an STR task using the proposed method is implemented with C++ language. It is shown that the proposed method has little precision loss compared with its floating-point counterpart. Our experiments demonstrate the compression ratio of the storage required by the proposed beam search decoding algorithm are 29.49 (ASR) and 17.95 (STR).

Visualization of Neural Network Predictions for Weather Forecasting

AbstractRecurrent neural networks are prime candidates for learning evolutions in multi‐dimensional time series data. The performance of such a network is judged by the loss function, which is aggregated into a scalar value that decreases during training. Observing only this number hides the variation that occurs within the typically large training and testing data sets. Understanding these variations is of highest importance to adjust network hyper‐parameters, such as the number of neurons, number of layers or to adjust the training set to include more representative examples. In this paper, we design a comprehensive and interactive system that allows users to study the output of recurrent neural networks on both the complete training data and testing data. We follow a coarse‐to‐fine strategy, providing overviews of annual, monthly and daily patterns in the time series and directly support a comparison of different hyper‐parameter settings. We applied our method to a recurrent convolutional neural network that was trained and tested on 25 years of climate data to forecast meteorological attributes, such as temperature, pressure and wind velocity. We further visualize the quality of the forecasting models, when applied to various locations on the Earth and we examine the combination of several forecasting models.