Class Of Recurrent Neural Networks Research Articles

Fake News Identification and Classification Using DSSM and Improved Recurrent Neural Network Classifier

ABSTRACTThe widespread use of social media has enormous consequences for the society, culture and business with potentially positive and negative effects. As online social networks are increasingly used for dissemination of information, at the same they are also becoming a medium for the spread of fake news for various commercial and political purposes. Technologies such as Artificial Intelligence (AI) and Natural Language Processing (NLP) tools offer great promise for researchers to build systems, which could automatically detect fake news. However, detecting fake news is a challenging task to accomplish as it requires models to summarize the news and compare it to the actual news in order to classify it as fake. This project proposes a framework that detects and classifies fake news messages using improved Recurrent Neural Networks and Deep Structured Semantic Model. The proposed approach intuitively identifies important features associated with fake news without previous domain knowledge while achieving accuracy 99%. The performance analysis method used for the proposed system is based on accuracy, specificity and sensitivity.

Recurrent neural networks for reconstructing complex directed brain connectivity.

While Granger Causality(GC)-based approaches have been widely employed in a vast number of problems in network science, the vast majority of GC applications are based on linear multivariate autoregressive (MVAR) models. However, it is well known that real-life system (and biological networks in particular) exhibit notable nonlinear behavior, hence undermining that validity of MVAR-based approaches to estimating GC (MVAR-GC). In this paper, we define a novel approach to estimating nonlinear, directed within-network interactions based on a specific class of recurrent neural networks (RNN) termed echo-state networks (ESN). We reformulate the classical GC framework in terms of ESN-based models for multivariate signals generated by arbitrarily complex networks, and characterize the ability of our ESN-based Granger Causality (ES-GC) to capture nonlinear causal relations by simulating multivariate coupling in a network of nonlinearly interacting, noisy Duffing oscillators operating in a chaotic regime. Synthetic validation shows a net advantage of ES-GC over all other estimators in detecting nonlinear, causal links. We then explore the structure of EC-GC networks in the human brain in functional MRI data from 1003 healthy subjects scanned at rest at 3T, discovering previously unknown between-network interactions. In summary, ES-GC performs significantly better than commonly used and recently developed GC detection tools, making it a superior tool for the analysis of e.g. multivariate biological networks.

In silico learning of tumor evolution through mutational time series

Cancer arises through the accumulation of somatic mutations over time. Understanding the sequence of mutation occurrence during cancer progression can assist early and accurate diagnosis and improve clinical decision-making. Here we employ long short-term memory (LSTM) networks, a class of recurrent neural network, to learn the evolution of a tumor through an ordered sequence of mutations. We demonstrate the capacity of LSTMs to learn complex dynamics of the mutational time series governing tumor progression, allowing accurate prediction of the mutational burden and the occurrence of mutations in the sequence. Using the probabilities learned by the LSTM, we simulate mutational data and show that the simulation results are statistically indistinguishable from the empirical data. We identify passenger mutations that are significantly associated with established cancer drivers in the sequence and demonstrate that the genes carrying these mutations are substantially enriched in interactions with the corresponding driver genes. Breaking the network into modules consisting of driver genes and their interactors, we show that these interactions are associated with poor patient prognosis, thus likely conferring growth advantage for tumor progression. Thus, application of LSTM provides for prediction of numerous additional conditional drivers and reveals hitherto unknown aspects of cancer evolution.

含分布时滞递归神经网络的一般衰减同步

The general decay synchronization (GDS) of a class of recurrent neural networks (RNNs) with general activation functions and distributed delays was studied. By means of suitable LyapunovKrasovskii functionals and useful inequality techniques, some sufficient conditions for the GDS of considered RNNs were established via a type of nonlinear control. An example with numerical simulations illustrates the correctness of the obtained theoretical results.

Toward Deep Learning Approaches for Learning Structure Motifs and Classifying Biological Sequences From RNA A-to-I Editing Events

RNA editing is a post-transcriptional RNA sequence modification that alters the mature RNA sequence from its template DNA sequences. RNA editing events are critical in various biological and biochemical mechanisms, and can expand the transcriptomic and proteomic diversity from altered gene regulation to mutations. A-to-I RNA editing is now being vastly detected and quantified on a global scale and gained much attention. A deeper understanding of this process with insufficient genomic annotations and prior knowledge-based filtering steps, such as high-throughput next-generation sequencing techniques, are needed, in addition to data regarding whether an editing location is located in one of the following three main classes of RNA editing sites: ALU in Alu repetitive elements, REP in non-Alu repetitive elements, and NONREP in non-repetitive regions. This study proposes deep learning approaches to ameliorate these issues by learning motif patterns and identifying regions of A-to-I editing events in biological sequences. Using datasets derived from the public REDI portal, 300,000 editing sites were equally divided and used for learning sequence and structure motifs visualized by convolutional kernels. We explored the RNA editing pattern changes using information of positional and class enrichment of learned motifs in the three aforementioned classes. We demonstrated that these newly investigated approaches using large-scale RNA sequencing data offer excellent classification accuracy with a well-optimized convolutional neural network and recurrent neural network classifiers that obtained average area under curves of 0.960 and 0.962, respectively. The findings further decipher the principles underlying RNA editing events and will facilitate more effective RNA sequencing research.

Classification of Widely and Rarely Expressed Genes with Recurrent Neural Network

A tissue-specific gene expression shapes the formation of tissues, while gene expression changes reflect the immune response of the human body to environmental stimulations or pressure, particularly in disease conditions, such as cancers. A few genes are commonly expressed across tissues or various cancers, while others are not. To investigate the functional differences between widely and rarely expressed genes, we defined the genes that were expressed in 32 normal tissues/cancers (i.e., called widely expressed genes; FPKM >1 in all samples) and those that were not detected (i.e., called rarely expressed genes; FPKM <1 in all samples) based on the large gene expression data set provided by Uhlen et al. Each gene was encoded using the gene ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) enrichment scores. Minimum redundancy maximum relevance (mRMR) was used to measure and rank these features on the mRMR feature list. Thereafter, we applied the incremental feature selection method with a supervised classifier recurrent neural network (RNN) to select the discriminate features for classifying widely expressed genes from rarely expressed genes and construct an optimum RNN classifier. The Youden's indexes generated by the optimum RNN classifier and evaluated using a 10-fold cross validation were 0.739 for normal tissues and 0.639 for cancers. Furthermore, the underlying mechanisms of the key discriminate GO and KEGG features were analyzed. Results can facilitate the identification of the expression landscape of genes and elucidation of how gene expression shapes tissues and the microenvironment of cancers.

Brain dynamics and temporal trajectories during task and naturalistic processing

Human functional Magnetic Resonance Imaging (fMRI) data are acquired while participants engage in diverse perceptual, motor, cognitive, and emotional tasks. Although data are acquired temporally, they are most often treated in a quasi-static manner. Yet, a fuller understanding of the mechanisms that support mental functions necessitates the characterization of dynamic properties. Here, we describe an approach employing a class of recurrent neural networks called reservoir computing, and show the feasibility and potential of using it for the analysis of temporal properties of brain data. We show that reservoirs can be used effectively both for condition classification and for characterizing lower-dimensional “trajectories” of temporal data. Classification accuracy was approximately 90% for short clips of “social interactions” and around 70% for clips extracted from movie segments. Data representations with 12 or fewer dimensions (from an original space with over 300) attained classification accuracy within 5% of the full data. We hypothesize that such low-dimensional trajectories may provide “signatures” that can be associated with tasks and/or mental states. The approach was applied across participants (that is, training in one set of participants, and testing in a separate group), showing that representations generalized well to unseen participants. Taken together, we believe the present approach provides a promising framework to characterize dynamic fMRI information during both tasks and naturalistic conditions.

Approximate Computing for Long Short Term Memory (LSTM) Neural Networks

Long Short Term Memory (LSTM) networks are a class of recurrent neural networks that are widely used for machine learning tasks involving sequences, including machine translation, text generation, and speech recognition. Large-scale LSTMs, which are deployed in many real-world applications, are highly compute intensive. To address this challenge, we propose AxLSTM, an application of approximate computing to improve the execution efficiency of LSTMs. An LSTM is composed of cells, each of which contains a cell state along with multiple gating units that control the addition and removal of information from the state. The LSTM execution proceeds in timesteps, with a new symbol of the input sequence processed at each timestep. AxLSTM consists of two techniques—Dynamic Timestep Skipping (DTS) and Dynamic State Reduction (DSR). DTS identifies, at runtime, input symbols that are likely to have little or no impact on the cell state and skips evaluating the corresponding timesteps. In contrast, DSR reduces the size of the cell state in accordance with the complexity of the input sequence, leading to a reduced number of computations per timestep. We describe how AxLSTM can be applied to the most common application of LSTMs, viz. , sequence-to-sequence learning. We implement AxLSTM within the TensorFlow deep learning framework and evaluate it on 3 state-of-the-art sequence-to-sequence models. On a 2.7 GHz Intel Xeon server with 128 GB memory and 32 processor cores, AxLSTM achieves $ {1.08\times -1.31 \times }$ speedups with minimal loss in quality, and $ {1.12 \times -1.37 \times }$ speedups when moderate reductions in quality are acceptable.

Pth moment exponential input‐to‐state stability of nonlinear discrete‐time impulsive stochastic delay systems

SummaryThis paper investigates the pth moment exponential input‐to‐state stability (ISS) of nonlinear discrete‐time impulsive stochastic delay systems. By employing Lyapunov functionals, some pth moment exponential ISS criteria are provided. The obtained results show that a discrete‐time stochastic delay system can become pth moment exponential input‐to‐state stable by impulsive controls even if it may be not input‐to‐state stable itself. On the other hand, the original system without impulses can retain its ISS property with appropriate destabilizing impulses. As an application, the theoretical results are used to test the ISS for a class of recurrent neural networks under stochastic perturbations. Finally, a numerical example is presented to illustrate the effectiveness of the results.

A Novel Similar Character Discrimination Method for Online Handwritten Urdu Character Recognition in Half Forms

Online handwritten Urdu character recognition is one of the key technologies for intelligent interface on smart phones and touch screens. It is a challenging research topic as Urdu script has many similar character groups. A novel similar character discrimination method for online handwritten Urdu character recognition is proposed in this paper which includes pre-classification, feature extraction and fine classification process. The pre-classifier enables the discrimination of similar characters by putting them in distinct smaller subsets according to stroke number and diacritics. Then structural features and wavelet features are extracted. Finally, Support Vector Machines (SVM), Artificial Neural Networks (ANN), and Recurrent Neural Network (RNN) classifiers are compared for fine classification within subsets. Results of RNN classifier without using the proposed pre-classifier and features have also been obtained to check the end-to-end capability of the RNN classifier. Experimental results show that the proposed method is efficient and achieves an overall accuracy of 96% on a large-scale self-collected dataset. It is feasible to extend this method for other Arabic scripts.

An Ensemble Learning Approach to the Predictive Stability of Echo State Networks

This paper demonstrates the inherent capacity of the ensemble learning approach to stabilize the notoriously difficult to tune echo state network (ESN) learning machines. ESN learning machines belong to a class of recurrent neural networks made popular because of their ability to perform predictively well on time series data. However, for any given task, building the corresponding optimal ESN is notoriously hard because of the instability of the tuning process. In this paper, we harness the high predictive variance by building ensembles made up of lightly tuned ESNs, and then combining those ESNs (base learners) rather than seeking the daunting task of selecting a single one. Throughout this paper, all our examples consistently demonstrate substantial reductions in average prediction error as a result of using the aggregate learning machine in place of the single one.

Inference of gene networks from gene expression time series using recurrent neural networks and sparse MAP estimation.

The inference of genetic regulatory networks (GRNs) provides insight into the cellular responses to signals. A class of recurrent neural networks (RNNs) capturing the dynamics of GRN has been used as a basis for inferring small-scale GRNs from gene expression time series. The Bayesian framework facilitates incorporating the hypothesis of GRN into the model estimation to improve the accuracy of GRN inference. We present new methods for inferring small-scale GRNs based on RNNs. The weights of wires of RNN represent the strengths of gene-to-gene regulatory interactions. We use a class of automatic relevance determination (ARD) priors to enforce the sparsity in the maximum a posteriori (MAP) estimates of wire weights of RNN. A particle swarm optimization (PSO) is integrated as an optimization engine into the MAP estimation process. Likely networks of genes generated based on estimated wire weights are combined using the majority rule to determine a final estimated GRN. As an alternative, a class of [Formula: see text] priors is used for attaining the sparse MAP estimates of wire weights of RNN. We also infer the GRN using the maximum likelihood (ML) estimates of wire weights of RNN. The RNN-based GRN inference algorithms, ARD-RNN, [Formula: see text] -RNN, and ML-RNN are tested on simulated and experimental E. coli and yeast time series containing 6-11 genes and 7-19 data points. Published GRN inference algorithms based on regressions and mutual information networks are performed on the benchmark datasets to compare performances. ARD and [Formula: see text] -norm priors are used for the estimation of wire weights of RNN. Results of GRN inference experiments show that ARD-RNN, [Formula: see text] -RNN have similar best accuracies on the simulated time series. The ARD-RNN is more accurate than [Formula: see text] -RNN, ML-RNN, and mostly more accurate than the reference algorithms on the experimental time series. The effectiveness of ARD-RNN for inferring small-scale GRNs using gene expression time series of limited length is empirically verified.

Online, Loudness-Invariant Vocal Detection in Mixed Music Signals

Singing voice detection, also referred to as vocal detection (VD), aims at automatically identifying the regions in a music recording where at least one person sings. It is highly challenging due to the timbral and expressive richness of the human singing voice, as well as the practically endless variety of interfering instrumental accompaniment. Additionally, certain instruments have an inherent risk of being misclassified as vocals due to similarities of the sound production system. In this paper, we present a machine learning approach that is based on our previous work for VD, which is specifically designed to deal with those challenging conditions. The contribution of this paper is threefold: First, we present a new method for VD that passes a compact set of features to a long short-term memory recurrent neural network classifier that obtains state-of-the-art results. Second, we thoroughly evaluate the proposed method along with related approaches to really probe the weaknesses of the methods. In order to allow for such a thorough evaluation, we make a curated collection of datasets available to the research community. Finally, we focus on a specific problem that was not obvious and had not been discussed in the literature so far. The reason for this is precisely because limited evaluations had not revealed this as a problem: the lack of loudness invariance. We will discuss the implications of utilizing loudness-related features and show that our method successfully deals with this problem due to the specific set of features it uses.

Effective hate-speech detection in Twitter data using recurrent neural networks

This paper addresses the important problem of discerning hateful content in social media. We propose a detection scheme that is an ensemble of Recurrent Neural Network (RNN) classifiers, and it incorporates various features associated with user-related information, such as the users' tendency towards racism or sexism. These data are fed as input to the above classifiers along with the word frequency vectors derived from the textual content. Our approach has been evaluated on a publicly available corpus of 16k tweets, and the results demonstrate its effectiveness in comparison to existing state of the art solutions. More specifically, our scheme can successfully distinguish racism and sexism messages from normal text, and achieve higher classification quality than current state-of-the-art algorithms.

Semi-Supervised Recurrent Neural Network for Adverse Drug Reaction mention extraction

BackgroundSocial media is a useful platform to share health-related information due to its vast reach. This makes it a good candidate for public-health monitoring tasks, specifically for pharmacovigilance. We study the problem of extraction of Adverse-Drug-Reaction (ADR) mentions from social media, particularly from Twitter. Medical information extraction from social media is challenging, mainly due to short and highly informal nature of text, as compared to more technical and formal medical reports.MethodsCurrent methods in ADR mention extraction rely on supervised learning methods, which suffer from labeled data scarcity problem. The state-of-the-art method uses deep neural networks, specifically a class of Recurrent Neural Network (RNN) which is Long-Short-Term-Memory network (LSTM). Deep neural networks, due to their large number of free parameters rely heavily on large annotated corpora for learning the end task. But in the real-world, it is hard to get large labeled data, mainly due to the heavy cost associated with the manual annotation.ResultsTo this end, we propose a novel semi-supervised learning based RNN model, which can leverage unlabeled data also present in abundance on social media. Through experiments we demonstrate the effectiveness of our method, achieving state-of-the-art performance in ADR mention extraction.ConclusionIn this study, we tackle the problem of labeled data scarcity for Adverse Drug Reaction mention extraction from social media and propose a novel semi-supervised learning based method which can leverage large unlabeled corpus available in abundance on the web. Through empirical study, we demonstrate that our proposed method outperforms fully supervised learning based baseline which relies on large manually annotated corpus for a good performance.

Delay-dependent and delay-independent passivity of a class of recurrent neural networks with impulse and multi-proportional delays

Abstract The problem for passivity of a class of recurrent neural networks (RNNs) with impulse and multi-proportional delays is investigated in this paper. Several delay-dependent and delay-independent passivity criteria for the given system are obtained by establishing appropriate Lyapunov–krasovskill functionals and applying matrix inequality approach. The passivity criteria here are presented in the form of linear matrix inequalities (LMIs), which can be easily verified by Matlab Toolbox. Simultaneously, the criteria obtained here include the passivity results of RNNs without impulse or without delays as special cases and can also be extended to other RNNs with more complicated impulsive noise. Finally, two numerical examples with some simulations shows that the proposed results are effective.

A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks.

To accommodate structured approaches of neural computation, we propose a class of recurrent neural networks for indexing and storing sequences of symbols or analog data vectors. These networks with randomized input weights and orthogonal recurrent weights implement coding principles previously described in vector symbolic architectures (VSA) and leverage properties of reservoir computing. In general, the storage in reservoir computing is lossy, and crosstalk noise limits the retrieval accuracy and information capacity. A novel theory to optimize memory performance in such networks is presented and compared with simulation experiments. The theory describes linear readout of analog data and readout with winner-take-all error correction of symbolic data as proposed in VSA models. We find that diverse VSA models from the literature have universal performance properties, which are superior to what previous analyses predicted. Further, we propose novel VSA models with the statistically optimal Wiener filter in the readout that exhibit much higher information capacity, in particular for storing analog data. The theory we present also applies to memory buffers, networks with gradual forgetting, which can operate on infinite data streams without memory overflow. Interestingly, we find that different forgetting mechanisms, such as attenuating recurrent weights or neural nonlinearities, produce very similar behavior if the forgetting time constants are matched. Such models exhibit extensive capacity when their forgetting time constant is optimized for given noise conditions and network size. These results enable the design of new types of VSA models for the online processing of data streams.

Text Mining Analysis of Teaching Evaluation Questionnaires for the Selection of Outstanding Teaching Faculty Members

This paper was conducted in collaboration with the Office of Institutional Research at National Ilan University in Taiwan to analyze textual opinions found in teaching evaluation questionnaires and applied the analysis results to assisting the selection of outstanding teaching faculty members. The selection of outstanding teachers requires that selection committee members spend a large amount of time reviewing written data. Therefore, this paper develops a set of systems for the analysis of textual opinions in teaching evaluation questionnaires, providing reference materials for the selection committee. The teaching evaluation questionnaire is a form of educational data. This paper analyzes these data using educational data mining. In text mining, text sentiment analysis is a common textual data quantification method that can analyze the sentiment tendency of a text author. This paper uses a text sentiment analysis to quantify the students’ textual opinions and to provide the selection committee with the sentiment tendency of students’ comments on teaching faculty members. We analyze text sentiment separately for different classifiers by using the Chinese text sentiment analysis kit SnowNLP. We compare the efficacy of classifiers that do not take time series factors into consideration (naive Bayes and fully connected neural network) to those that do [recurrent neural network (RNN), long short-term memory (LSTM) RNN, and attention RNN]. We found that classifiers that consider time series factors are more effective at analyzing text sentiment. Furthermore, adding LSTM cells and an attention mechanism to a traditional RNN classifier effectively improved its efficacy on long-sequence tasks. As a result, we chose the attention LSTM classifier—with a positive sentiment recognition rate of 97% and a negative sentiments recognition rate of 87%—as our preferred text sentiment classifier. Finally, we set up an analytics server that will be modularized to facilitate its integration into the systems of different schools.

Automatic Detection of Compensation During Robotic Stroke Rehabilitation Therapy.

Robotic stroke rehabilitation therapy can greatly increase the efficiency of therapy delivery. However, when left unsupervised, users often compensate for limitations in affected muscles and joints by recruiting unaffected muscles and joints, leading to undesirable rehabilitation outcomes. This paper aims to develop a computer vision system that augments robotic stroke rehabilitation therapy by automatically detecting such compensatory motions. Nine stroke survivors and ten healthy adults participated in this study. All participants completed scripted motions using a table-top rehabilitation robot. The healthy participants also simulated three types of compensatory motions. The 3-D trajectories of upper body joint positions tracked over time were used for multiclass classification of postures. A support vector machine (SVM) classifier detected lean-forward compensation from healthy participants with excellent accuracy (AUC = 0.98, F1 = 0.82), followed by trunk-rotation compensation (AUC = 0.77, F1 = 0.57). Shoulder-elevation compensation was not well detected (AUC = 0.66, F1 = 0.07). A recurrent neural network (RNN) classifier, which encodes the temporal dependency of video frames, obtained similar results. In contrast, F1-scores in stroke survivors were low for all three compensations while using RNN: lean-forward compensation (AUC = 0.77, F1 = 0.17), trunk-rotation compensation (AUC = 0.81, F1 = 0.27), and shoulder-elevation compensation (AUC = 0.27, F1 = 0.07). The result was similar while using SVM. To improve detection accuracy for stroke survivors, future work should focus on predefining the range of motion, direct camera placement, delivering exercise intensity tantamount to that of real stroke therapies, adjusting seat height, and recording full therapy sessions.

Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

We present a model of pragmatic referring expression interpretation in a grounded communication task (identifying colors from descriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recursive pragmatic reasoning framework. Experiments show that this combined pragmatic model interprets color descriptions more accurately than the classifiers from which it is built, and that much of this improvement results from combining the speaker and listener perspectives. We observe that pragmatic reasoning helps primarily in the hardest cases: when the model must distinguish very similar colors, or when few utterances adequately express the target color. Our findings make use of a newly-collected corpus of human utterances in color reference games, which exhibit a variety of pragmatic behaviors. We also show that the embedded speaker model reproduces many of these pragmatic behaviors.