Abstract
Reservoir computers (RCs) and recurrent neural networks (RNNs) can mimic any finite-state automaton in theory, and some workers demonstrated that this can hold in practice. We test the capability of generalized linear models, RCs, and Long Short-Term Memory (LSTM) RNN architectures to predict the stochastic processes generated by a large suite of probabilistic deterministic finite-state automata (PDFA) in the small-data limit according to two metrics: predictive accuracy and distance to a predictive rate-distortion curve. The latter provides a sense of whether or not the RNN is a lossy predictive feature extractor in the information-theoretic sense. PDFAs provide an excellent performance benchmark in that they can be systematically enumerated, the randomness and correlation structure of their generated processes are exactly known, and their optimal memory-limited predictors are easily computed. With less data than is needed to make a good prediction, LSTMs surprisingly lose at predictive accuracy, but win at lossy predictive feature extraction. These results highlight the utility of causal states in understanding the capabilities of RNNs to predict.
Highlights
We focus on three methods for time series prediction: generalized linear models (GLM), reservoir computers (RCs), and Long Short-Term Memory (LSTM)
An aim here is to thoroughly and systematically analyze the predictive accuracy as measured by the probability of correctly guessing the symbol and code rate of our three time series predictors of a large swath of probabilistic deterministic finite-state automata (PDFA) in the small-data limit, in which only 5000 samples are shown to the recurrent neural networks (RNNs)
We find that all three prediction methods–GLMs, RCs, and LSTMs—
Summary
Traders try to predict if a stock price will go up or down, adjusting investment strategies . Self-driving cars must predict the motion of other objects on and off the road. When it comes to biology, evidence suggests that organisms endeavor to predict their environment as a key survival strategy [1–3]. (This is “determinism” in the sense of formal language theory [30]—an automaton deterministically recognizes a string—not in the sense of nonstochastic. It was originally called unifilarity in the information theoretic analysis of hidden Markov chains [11]. PDFAs are known as unifilar hidden Markov models [12].)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
