Recurrent Neural Net Research Articles

The Application of Deep Learning Methods in the Forecasting and Imputation of Solar Irradiance Data

Accurate solar radiation forecasts are necessary for solar power plants and local energy grids. Precise solar predictions can optimise plant operations and assist in managing power generation fluctuations. High-quality solar data are required to perform solar forecasting and may be accessed through open databases such as the Baseline Surface Radiation Network. Due to maintenance and malfunctioning equipment, these databases may have missing data. Deep learning methods can extract relevant features and non-linearities in data, which make them suitable for forecasting and imputation tasks. The development of the transformer architecture in 2017 marked an improvement from recurrent sequence processing using RNN (recurrent neural net) architectures to parallel sequence handling using self-attention mechanisms. Previous deep learning irradiance forecasting models utilised a RNN or RNN extension architecture. This paper presents novel applications of Transformer models to solar imputation and forecasting. Two deep learning architectures – Transformer and SAITS (Self-Attention-based Imputation for Time Series) – were trained to impute time series of global horizontal irradiance. Additionally, a Temporal Fusion Transformer (TFT) deep learning architecture was trained to forecast future values of global horizontal irradiance time series. Feature concatenation allows for the combination of differing exogenous variables into a single input vector for a deep learning solar forecasting model. Previous solar forecasting models do not incorporate cloud data or otherwise utilise an exogenous simple categorical mask to indicate cloud cover. In this paper, GOES-16 cloud optical depth and particle size datasets provide an improved representation of cloud properties. Classical imputation and forecasting models were also developed and used as baselines for comparison with deep learning models. The models were evaluated using standard statistical metrics for time series. It was found that the deep learning TFT forecasting model produced better forecasts than the classical regression model. However, the classical KNN imputer performed better than the two deep learning imputation models.

신호 지시원의 경광봉을 사용한 수신호 인식 알고리즘 개발

In this paper, an algorithm that recognizes wand signals for traffic control is proposed. The Society of Automotive Engineers (SAE) defines 6 levels of driving automation ranging from 0 (fully manual) to 5 (fully autonomous). Starting from the level 4, the driver does not operate the driving systems. At this level of automation, the artificial intelligence needs to take control even in the most unexpected situations such as malfunctioning traffic signals, driveway impaired by car accidents, or passing through construction sites. In such situations, police or traffic safety foreman may take control of the site and command the autonomous vehicles to slow down, stop or go. This paper presents a computer vision-based algorithm to detect and recognize the traffic control wand signals. The algorithm is composed of tracking the foreman (officer), detecting the wand, and classifying the action of the foreman based on wand trajectory by using Recurrent Neural Net(RNN) based networks.

P6568Forecasting atrial fibrillation using machine learning techniques

Abstract Background Forecasting atrial fibrillation (AF) a few minutes before its onset has been studied, mainly based on heart rate variability parameters, derived from 24-hour ECG Holter monitorings. However, these studies have shown conflicting, non-clinically applicable results. Nowadays, machine learning algorithms have proven their ability to anticipate events. Therefore, forecasting AF before its onset should be (re)assessed using machine learning techniques. A reliable forecasting could improve results of preventive pacing in patients with cardiac electronic implanted devices (CEID). Purpose To forecast an oncoming AF episode in individual patients using machine learning techniques. To evaluate the effect if the onset of an AF episode can be forecasted on longer time frames. Methods The totality of the raw data of a data set of 10484 ECG Holter monitorings was retrospectively analyzed and all AF episodes were annotated. Onset of each AF episode was determined with a precision of 5 msec. We only took AF events into consideration if they lasted longer than 30 seconds. Of all patients in the dataset, 140 presented paroxysmal AF (286 recorded AF episodes). We only used RR intervals to predict the presence of AF. We developed two different types of machine learning algorithms with different computational power requirements: a “dynamic” deep and recurrent neural net (RNN) and a “static” decision-tree with adaboost (boosting trees) more suitable for embedded devices. These algorithms were trained on one set of patients (around 90%) and tested on the remaining set of patients (around 10%). Results The performance figures are summarized in the table. Both algorithms can be tuned to increase their specificity (at a loss of sensitivity) or vice versa, depending on the objective. Performance of forecasting algorithms RR-distance Boosting trees AUC RNN AUC 30–1 RR-Interval(s) before an AF event 97.1% 98.77% 60–31 RR-Intervals before an AF event 97.5% 99.1% 90–61 RR-Intervals before an AF event 96.9% 99.1% 120–91 RR-Inervals before an AF event 98.2% 98.9% AUC for Area Under ROC Curves. Conclusion Based upon this retrospective study, we show that AF can be forecasted on an individual level with high predictive power using machine learning algorithm, with little drop-off of predictive value within the studied distances (1–120 RR intervals before a potential AF episode). We believe that the embedding of our new algorithm(s) in CEID's could open the way to innovative therapies that significantly decrease AF burden in selected implanted patients.

Modeling Interval Timing by Recurrent Neural Nets.

The purpose of this study was to take a new approach in showing how the central nervous system might encode time at the supra-second level using recurrent neural nets (RNNs). This approach utilizes units with a delayed feedback, whose feedback weight determines the temporal properties of specific neurons in the network architecture. When these feedback neurons are coupled, they form a multilayered dynamical system that can be used to model temporal responses to steps of input in multidimensional systems. The timing network was implemented using separate recurrent “Go” and “No-Go” neural processing units to process an individual stimulus indicating the time of reward availability. Outputs from these distinct units on each time step are converted to a pulse reflecting a weighted sum of the separate Go and No-Go signals. This output pulse then drives an integrator unit, whose feedback weight and input weights shape the pulse distribution. This system was used to model empirical data from rodents performing in an instrumental “peak interval timing” task for two stimuli, Tone and Flash. For each of these stimuli, reward availability was signaled after different times from stimulus onset during training. Rodent performance was assessed on non-rewarded trials, following training, with each stimulus tested individually and simultaneously in a stimulus compound. The associated weights in the Go/No-Go network were trained using experimental data showing the mean distribution of bar press rates across an 80 s period in which a tone stimulus signaled reward after 5 s and a flash stimulus after 30 s from stimulus onset. Different Go/No-Go systems were used for each stimulus, but the weighted output of each fed into a final recurrent integrator unit, whose weights were unmodifiable. The recurrent neural net (RNN) model was implemented using Matlab and Matlab’s machine learning tools were utilized to train the network using the data from non-rewarded trials. The neural net output accurately fit the temporal distribution of tone and flash-initiated bar press data. Furthermore, a “Temporal Averaging” effect was also obtained when the flash and tone stimuli were combined. These results indicated that the system combining tone and flash responses were not superposed as in a linear system, but that there was a non-linearity, which interacted between tone and flash. In order to achieve an accurate fit to the empirical averaging data it was necessary to implement non-linear “saliency functions” that limited the output signal of each stimulus to the final integrator when the other was co-present. The model suggests that the central nervous system encodes timing generation as a dynamical system whose timing properties are embedded in the connection weights of the system. In this way, event timing is coded similar to the way other sensory-motor systems, such as the vestibulo-ocular and optokinetic systems, which combine sensory inputs from the vestibular and visual systems to generate the temporal aspects of compensatory eye movements.

Exponential Language Modeling Using Morphological Features and Multi-Task Learning

For languages with fast vocabulary growth and limited resources, data sparsity leads to challenges in training a language model. One strategy for addressing this problem is to leverage morphological structure as features in the model. This paper explores different uses of unsupervised morphological features in both the history and prediction space for three word-based exponential models (maximum entropy, logbilinear, and recurrent neural net (RNN)). Multi-task training is introduced as a regularizing mechanism to improve performance in the continuous-space approaches. The models are compared to non-parametric baselines. From using the RNN with morphological features and multi-task learning, experiments with conversational speech from four languages show we can obtain consistent gains of 7–11% in perplexity reduction in a limited-resource scenario (10 hrs speech), and 12–18% when the training size is increased ( 80 hrs ). Results are mixed for all other approaches, compared to a modified Kneser-Ney baseline, but morphology is useful in continuous-space models compared to their word-only baseline. Multi-task learning improves both continuous-space models.

Self-organization of behavioral primitives as multiple attractor dynamics: A robot experiment

This paper investigates how behavior primitives are self-organized in a neural network model utilizing a distributed representation scheme. The model is characterized by so-called parametric biases which adaptively modulate the encoding of different behavior patterns in a single recurrent neural net (RNN). Our experiments, using a real robot arm, showed that a set of end-point and oscillatory behavior patterns are learned by self-organizing fixed points and limit cycle dynamics that form behavior primitives. It was also found that diverse novel behavior patterns can be generated by modulating the parametric biases arbitrarily. Our analysis showed that such diversity in behavior generation emerges because a nonlinear map is self-organized between the space of parametric biases and that of the behavior patterns. The origin of the observed nonlinearity from the distributed representation is discussed. This paper investigates how behavior primitives are self-organized in a neural network model utilizing a distributed representation scheme. Our robot experiments showed that a set of end-point and oscillatory behavior patterns are learned by self-organizing fixed points and limit cycle dynamics that form behavior primitives. It was also found that diverse novel behavior patterns, in addition to previously learned patterns, can be generated by taking advantage of nonlinear effects that emerge from the distributed representation.

Recurrent learning algorithms for designing optimal controllers of continuous systems

Proposes a recurrent learning algorithm for designing the controllers of continuous dynamical systems in optimal control problems. The controllers are in the form of unfolded recurrent neural nets embedded with physical laws from classical control techniques. The learning algorithm is characterized by a double forward-recurrent-loops structure for solving both temporal recurrent and structure recurrent problems. The first problem results from the nature of general optimal control problems, where the objective functions are often related to (evaluated at) some specific time steps or system states only, causing missing learning signals at some steps or states. The second problem is due to the high-order discretization of continuous systems by the Runge-Kutta method that we perform to increase accuracy. This discretization transforms the system into several identical interconnected subnetworks, like a recurrent neural net expanded in the time axis. Two recurrent learning algorithms with different convergence properties are derived; first- and second-order learning algorithms. Their computations are local and performed efficiently as net signal propagation. We also propose two new nonlinear control structures for the 2D guidance problem and the optimal PI control problem. Under the training of the recurrent learning algorithms, these controllers can be easily tuned to be suboptimal for given objective functions. Extensive computer simulations show the controllers' optimization and generalization abilities.

Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems

This paper describes how agents can learn an internal model of the world structurally by focusing on the problem of behavior-based articulation. We develop an on-line learning scheme—the so-called mixture of recurrent neural net (RNN) experts—in which a set of RNN modules become self-organized as experts on multiple levels, in order to account for the different categories of sensory-motor flow which the robot experiences. Autonomous switching of activated modules in the lower level actually represents the articulation of the sensory-motor flow. In the meantime, a set of RNNs in the higher level competes to learn the sequences of module switching in the lower level, by which articulation at a further, more abstract level can be achieved. The proposed scheme was examined through simulation experiments involving the navigation learning problem. Our dynamical system analysis clarified the mechanism of the articulation. The possible correspondence between the articulation mechanism and the attention switching mechanism in thalamo-cortical loops is also discussed.

Dynamic On-line Clustering and State Extraction: An Approach to Symbolic Learning

Although recurrent neural nets have been moderately successful in learning to emulate finite-state machines (FSMs), the continuous internal state dynamics of a neural net are not well matched to the discrete behavior of an FSM. We describe an architecture, called DOLCE, that allows discrete states to evolve in a net as learning progresses. DOLCE consists of a standard recurrent neural net trained by gradient descent and an adaptive clustering technique that quantizes the state space. We describe two implementations of DOLCE. The first implementation, called DOLCE u , uses an adaptive clustering scheme in an unsupervised mode to determine both the number of clusters and the partitioning of the state space as learning progresses. The second model, DOLCE s , uses a Gaussian Mixture Model in a supervised learning framework to infer the states of an FSM. DOLCE s is based on the assumption that a finite set of discrete internal states is required for the task, and that the actual network state belongs to this set but has been corrupted by noise due to inaccuracy in the weights. DOLCE s learns to recover the discrete state with maximum a posteriori probability from the noisy state. Simulations show that both implementations of DOLCE lead to a significant improvement in generalization performance over earlier neural net approaches to FSM induction. The idea of adaptive quantization is not just applicable to DOLCE but can be applied to other domains as well.