Stable Learning Research Articles

Stable reinforcement learning via temporal competition between LTP and LTD traces

Neuronal systems that are involved in reinforcement learning must solve the temporal credit assignment problem, i.e., how is a stimulus associated with a reward that is delayed in time? Theoretical studies [1-3] have postulated that neural activity underlying learning ‘tags’ synapses with an ‘eligibility trace’, and that the subsequent arrival of a reward converts the eligibility traces into actual modification of synaptic efficacies. While eligibility traces provide one simple solution to the temporal credit assignment problem, they alone do not constitute a stable learning rule because there is no other mechanism indicating when learning should cease. In order to attain stability, rules involving eligibility traces often assume that once the association is learned, further learning is prevented via an inhibition of the reward stimulus [1,3,4]. Although synaptic plasticity is responsible for reinforcement learning in the brain, theories of reinforcement learning are generally abstract and involve neither neurons nor synapses. Furthermore, biophysical theories of synaptic plasticity typically model unsupervised learning and ignore the contribution of reinforcement. Here we describe a biophysically based theory of reinforcementmodulated synaptic plasticity and postulate the existence of two eligibility traces with different temporal profiles: one corresponding to the induction of LTP, and the other to the induction of LTD. The traces have different kinetics and their difference in magnitude at the time of reward determines if synaptic modification will correspond to LTP or LTD. Due to the difference in their decay rates, the LTP and LTD traces can exhibit temporal competition at the reward time and thus provides a mechanism for stable reinforcement learning without the need to inhibit reward. We test this novel reinforcement-learning rule on an experimentally motivated model of a recurrent cortical network [5], and compare the model results to experimental results at both the cellular and circuit levels. We further suggest that these eligibility traces are implemented via kinases and phosphatases, thus accounting for results at both the cellular and system levels.

Décompression du canal carpien par chirurgie ultra-mini-invasive sous guidage échographique : une étude pilote externe

BackgroundAuthors have reported better outcomes, by reducing surgical dissection for carpal tunnel syndromes requiring surgery. Recently, a new sonographically guided technique for ultra-minimally invasive (ultra-MIS) carpal tunnel release (CTR) through 1mm incision has been described. HypothesisWe hypothesized that a clinical trial for comparing ultra-MIS versus mini-open carpal tunnel release (mini-OCTR) was feasible. Materials and methodsTo test our hypothesis, we conducted a pilot study for studying ultra-MIS versus mini-OCTR respectively performed through a 1mm or a 2cm incision. We defined success if primary feasibility objectives (safety and efficacy) as well as secondary feasibility objectives (recruitment rates, compliance, completion, treatment blinding, personnel resources and sample size calculation for the clinical trial) could be matched. Score for Quick-DASH questionnaire at final follow-up was studied as the primary variable for the clinical trial. Turnover times were studied for assessing learning curve stability. ResultsForty patients were allotted. Primary and secondary feasibility objectives were matched with the following occurrences: 70.2 % of eligible patients finally recruited; 4.2 % of randomization refusals; 26.6 patients/month recruited; 100 % patients receiving a blinded treatment; 97.5 % compliance and 100 % completion. A sample size of 91 patients was calculated for clinical trial validation. At final follow-up, preliminary results for Quick-Dash substantially favored ultra-MIS over mini-OCTR (average 14.54 versus 7.39) and complication rates were lower for ultra-MIS (5 % versus 20 %). A stable learning curve was observed for both groups. ConclusionsThe clinical trial is feasible. There is currently no evidence to contraindicate nor withhold the use of ultra-MIS for CTR. Level of evidenceIII.

Ultra minimally invasive sonographically guided carpal tunnel release: An external pilot study

BackgroundAuthors have reported better outcomes, by reducing surgical dissection for carpal tunnel syndromes requiring surgery. Recently, a new sonographically guided technique for ultra minimally invasive (Ultra-MIS) carpal tunnel release (CTR) through 1mm incision has been described. HypothesisWe hypothesized that a clinical trial for comparing Ultra-MIS versus Mini-open Carpal Tunnel Release (Mini-OCTR) was feasible. Materials and methodsTo test our hypothesis, we conducted a pilot study for studying Ultra-MIS versus Mini-OCTR respectively performed through a 1mm or a 2cm incision. We defined success if primary feasibility objectives (safety and efficacy) as well as secondary feasibility objectives (recruitment rates, compliance, completion, treatment blinding, personnel resources and sample size calculation for the clinical trial) could be matched. Score for Quick-DASH questionnaire at final follow-up was studied as the primary variable for the clinical trial. Turnover times were studied for assessing learning curve stability. ResultsForty patients were allotted. Primary and secondary feasibility objectives were matched with the following occurrences: 70.2% of eligible patients finally recruited; 4.2% of randomization refusals; 26.6 patients/month recruited; 100% patients receiving a blinded treatment; 97.5% compliance and 100% completion. A sample size of 91 patients was calculated for clinical trial validation. At final follow-up, preliminary results for Quick-Dash substantially favored Ultra-MIS over Mini-OCTR (average 14.54 versus 7.39) and complication rates were lower for Ultra-MIS (5% versus 20%). A stable learning curve was observed for both groups. ConclusionsThe clinical trial is feasible. There is currently no evidence to contraindicate nor withhold the use of Ultra-MIS for CTR. Level of evidenceIII.

Coexistence of Reward and Unsupervised Learning During the Operant Conditioning of Neural Firing Rates

A fundamental goal of neuroscience is to understand how cognitive processes, such as operant conditioning, are performed by the brain. Typical and well studied examples of operant conditioning, in which the firing rates of individual cortical neurons in monkeys are increased using rewards, provide an opportunity for insight into this. Studies of reward-modulated spike-timing-dependent plasticity (RSTDP), and of other models such as R-max, have reproduced this learning behavior, but they have assumed that no unsupervised learning is present (i.e., no learning occurs without, or independent of, rewards). We show that these models cannot elicit firing rate reinforcement while exhibiting both reward learning and ongoing, stable unsupervised learning. To fix this issue, we propose a new RSTDP model of synaptic plasticity based upon the observed effects that dopamine has on long-term potentiation and depression (LTP and LTD). We show, both analytically and through simulations, that our new model can exhibit unsupervised learning and lead to firing rate reinforcement. This requires that the strengthening of LTP by the reward signal is greater than the strengthening of LTD and that the reinforced neuron exhibits irregular firing. We show the robustness of our findings to spike-timing correlations, to the synaptic weight dependence that is assumed, and to changes in the mean reward. We also consider our model in the differential reinforcement of two nearby neurons. Our model aligns more strongly with experimental studies than previous models and makes testable predictions for future experiments.

Multi-model adaptive control based on fuzzy neural networks

In practical applications, especially in the industrial field, some uncertain factors often lead abrupt changes of the system parameters. In this paper, multi-model adaptive control (MMAC) based on fuzzy neural networks (FNN) is suggested to identify and control discrete-time nonlinear systems in such environment. Stable learning algorithms for fuzzy neural networks which are robust to any bounded uncertainty are applied during the identification and adaptive control. The procedure of MMAC based on different model sets and index function is given, and the stability of MMAC is proved. This kind of MMAC based on FNN can improve the transient response greatly in case of the stability of training process is guaranteed. The simulation results show that the control performance is better than traditional adaptive controllers when the system parameters have changed abruptly.

Trade-off between learning and exploitation: The Pareto-optimal versus evolutionarily stable learning schedule in cumulative cultural evolution

Inheritance of culture is achieved by social learning and improvement is achieved by individual learning. To realize cumulative cultural evolution, social and individual learning should be performed in this order in one’s life. However, it is not clear whether such a learning schedule can evolve by the maximization of individual fitness. Here we study optimal allocation of lifetime to learning and exploitation in a two-stage life history model under a constant environment. We show that the learning schedule by which high cultural level is achieved through cumulative cultural evolution is unlikely to evolve as a result of the maximization of individual fitness, if there exists a trade-off between the time spent in learning and the time spent in exploiting the knowledge that has been learned in earlier stages of one’s life. Collapse of a fully developed culture is predicted by a game-theoretical analysis where individuals behave selfishly, e.g., less learning and more exploiting. The present study suggests that such factors as group selection, the ability of learning-while-working (“on the job training”), or environmental fluctuation might be important in the realization of rapid and cumulative cultural evolution that is observed in humans.

Neural Network Model Reference Adaptive System Speed Estimation for Sensorless Control of a Doubly Fed Induction Generator

This article proposes a neural network model reference adaptive system for the rotor angle and speed estimation of the doubly fed induction generator used in wind turbines. The model reference adaptive system reference signal is the measured rotor current. The adaptive neural network adjusts the weights minimizing the rotor current vector squared error using the steepest descent algorithm. The neural network maximum stable learning rate will be determined for this application. The validity of the proposed neural network model reference adaptive system is verified and analyzed in a real prototype of 7.5-kW doubly fed induction generator. To validate the proposed estimator, the estimated rotor angle and speed in the process of connecting the doubly fed induction generator to the grid and the sensorless regulation according to a random wind speed profile are presented.

A Novel Wavelet-Neural-Network-Based Robust Controller for IPM Motor Drives

This paper presents a novel wavelet-neural-network (NN) (WNN)-based speed controller for interior permanent-magnet (IPM) motor drives. The speed error and its change are used as inputs to the proposed controller. The WNN controller parameters such as weights, translation, and dilation of the wavelet functions are regulated adaptively with speed and load. The back-propagation-based training algorithm is used for updating the WNN parameters. The Lyapunov stability is used for stable learning and robustness of the WNN controller. The IPM motor is operated above the rated speed using the flux-weakening control. The maximum-torque-per-ampere control is used below the rated speed. The proposed WNN-controller-based IPM drive is implemented using the MATLAB/Simulink software and the dSPACE digital signal processor hardware. The performance of the WNN controller is compared to those of the conventional proportional–integral (PI)-, PI–derivative-, and NN-based speed controllers. The WNN controller is found more robust and quicker than conventional controllers.

Cognitive Strategies of Encoding, Storage, and Retrieval of Lexicon Popular Techniques Applied by Iranian French Language Learners

This paper aims to present the results obtained in a survey whose objective was to investigate the Iranian French language learner’s definition of memorization, and the techniques they adopt most frequently in memorizing linguistic structures and vocabulary. The results indicate that although memorization plays a significant role in learning a foreign language, it is often neglected or understated. Memorization techniques are known to be rather individual, but it is suggested that weak learners can be encouraged to replace old techniques with more effective ones that could pave the way to a profound and stable learning.

An Improved Reinforcement Learning System Using Affective Factors

As a powerful and intelligent machine learning method, reinforcement learning (RL) has been widely used in many fields such as game theory, adaptive control, multi-agent system, nonlinear forecasting, and so on. The main contribution of this technique is its exploration and exploitation approaches to find the optimal solution or semi-optimal solution of goal-directed problems. However, when RL is applied to multi-agent systems (MASs), problems such as “curse of dimension”, “perceptual aliasing problem”, and uncertainty of the environment constitute high hurdles to RL. Meanwhile, although RL is inspired by behavioral psychology and reward/punishment from the environment is used, higher mental factors such as affects, emotions, and motivations are rarely adopted in the learning procedure of RL. In this paper, to challenge agents learning in MASs, we propose a computational motivation function, which adopts two principle affective factors “Arousal” and “Pleasure” of Russell’s circumplex model of affects, to improve the learning performance of a conventional RL algorithm named Q-learning (QL). Compared with the conventional QL, computer simulations of pursuit problems with static and dynamic preys were carried out, and the results showed that the proposed method results in agents having a faster and more stable learning performance.

Probabilistic computation underlying sequence learning in a spiking attractor memory network

Many cognitive and motor functions are enabled by the temporal representation and processing of stimuli but it remains an open issue how neuronal circuits could reliably encode such sequences of information. We consider the task of generating and learning spatiotemporal spike patterns in the context of an attractor memory network, in which each memory is stored in a distributed fashion represented by increased firing in pools of excitatory neurons. Excitatory activity is locally modulated by inhibitory neurons representing lateral inhibition that generates a type of winner-take-all dynamics. Networks of this type have previously been shown to exhibit switching between a non-coding ground state and low-rate memory state activations displaying gamma oscillations [1]; however, stable sequential associations between different attractors were not present. Assuming a probabilistic framework in which local neuron populations discretely encode uncertainty about an attribute in the external world (e.g. a column in visual cortex tuned to a specific edge orientation), we model inter-module synapses using the Bayesian Confidence Propagation Neural Network (BCPNN) plasticity rule [2]. We use a spike-based version of BCPNN in which synaptic weights are statistically inferred by estimating the posterior likelihood of activation for the postsynaptic cell upon presentation of evidence in the form of presynaptic activity patterns. Probabilities are estimated on-line using local exponentially weighted moving averages, with time scales that are biologically motivated by the cascade of events involved in the induction and maintenance of long-term plasticity. Modulating the kinetics of these traces is shown to shape the width of the STDP kernel, which in turn allows attractors to be learned forwards or backwards through time. Stable learning is confirmed by aunimodal stationary weight distribution. Inference additionally requires modification of a distinct neuronal component, which we interpret as a correlate of intrinsic excitability. Such synaptic [3] and nonsynaptic [4] mechanisms were specifically shown to be relevant for learning and inference. In broader terms, our model instead suggests the presence of and interaction between all of these processes in approximating Bayesian computation. Introducing plastic BCPNN synaptic projections into the attractor network model allows for stable associations between distinct network states. Associations are mediated by different synaptic timescales [5] with fast (AMPA type) and slower (NMDA type) dynamics that in conjunction with the spiking BCPNN rule produce sequences of attractor activations. We demonstrate the feasibility of our model using network simulations of integrate-and-fire neurons, and find that the ability to learn sequences depends on the specific structure of the inhibitory microcircuitry and on the local balance of excitation and inhibition in the network. Preliminary results show that the network can reliably store spatiotemporal patterns consisting of hundreds of discrete network states using just a few thousand neurons. Moreover, excitatory pools can participate multiple times in the sequence, suggesting that spiking attractor networks of this type could support an efficient combinatorial code. Our model provides novel insights into how local and global computations found throughout neocortex and hippocampus, framed in the context of probabilistic inference, could contribute to generating and learning sequential neural activity.

ART2 알고리즘을 이용한 효율적인 스마트폰 어플리케이션 실행 방법

With probably too many smart phone applications downloaded, it is sometimes frustrating to find frequently used applications quickly. In this paper, we propose a learning application that learns what applications a user frequently uses and match them with several signals that user designated. This learning applications uses ART2 algorithm that is good for stable real-time learning. By executing this learning application, a user simply chooses an application that is to be quickly searched and then draw a figure that would match the designated application at the upper left corner of our learning application. The proposed learning application transforms the background with 0`s and the figure with 1`s and normalize them to be used as inputs for ART2 and ART2 does clustering to setup a match table between applications and figures. After learning, a user simply draws a figure to execute one`s frequently used application.

Robust on-line nonlinear systems identification using multilayer dynamic neural networks with two-time scales

This paper presents a novel identification method for nonlinear systems including the aspects of fast and slow phenomenon via dynamic multilayer neural networks (NN) with two-time scales. The Lyapunov function and singularly perturbed techniques are used to develop the stable learning procedures for the hidden layers and output layers of the dynamic neural networks model. The proposed learning algorithm is similar to the well-known propagation rule of the multilayer perceptrons but with the novel correction terms which guarantee bounded tracking errors and bounded weights. The passivity approach is used to prove that the proposed neural identifier is robust and avoids the need for persistently exciting (PE) conditions. The effectiveness of the algorithm is illustrated via simulation of an electric DC motor and an induction motor identification.

ON OPTIMAL LEARNING SCHEDULES AND THE MARGINAL VALUE OF CUMULATIVE CULTURAL EVOLUTION

The age-dependent choice between expressing individual learning (IL) or social learning (SL) affects cumulative cultural evolution. A learning schedule in which SL precedes IL is supportive of cumulative culture because the amount of nongenetically encoded adaptive information acquired by previous generations can be absorbed by an individual and augmented. Devoting time and energy to learning, however, reduces the resources available for other life-history components. Learning schedules and life history thus coevolve. Here, we analyze a model where individuals may have up to three distinct life stages: "infants" using IL or oblique SL, "juveniles" implementing IL or horizontal SL, and adults obtaining material resources with learned information. We study the dynamic allocation of IL and SL within life stages and how this coevolves with the length of the learning stages. Although no learning may be evolutionary stable, we find conditions where cumulative cultural evolution can be selected for. In that case, the evolutionary stable learning schedule causes individuals to use oblique SL during infancy and a mixture between IL and horizontal SL when juvenile. We also find that the selected pattern of oblique SL increases the amount of information in the population, but horizontal SL does not do so.

Stable learning of functional maps in self-organizing spiking neural networks with continuous synaptic plasticity

This study describes a spiking model that self-organizes for stable formation and maintenance of orientation and ocular dominance maps in the visual cortex (V1). This self-organization process simulates three development phases: an early experience-independent phase, a late experience-independent phase and a subsequent refinement phase during which experience acts to shape the map properties. The ocular dominance maps that emerge accommodate the two sets of monocular inputs that arise from the lateral geniculate nucleus (LGN) to layer 4 of V1. The orientation selectivity maps that emerge feature well-developed iso-orientation domains and fractures. During the last two phases of development the orientation preferences at some locations appear to rotate continuously through ±180° along circular paths and referred to as pinwheel-like patterns but without any corresponding point discontinuities in the orientation gradient maps. The formation of these functional maps is driven by balanced excitatory and inhibitory currents that are established via synaptic plasticity based on spike timing for both excitatory and inhibitory synapses. The stability and maintenance of the formed maps with continuous synaptic plasticity is enabled by homeostasis caused by inhibitory plasticity. However, a prolonged exposure to repeated stimuli does alter the formed maps over time due to plasticity. The results from this study suggest that continuous synaptic plasticity in both excitatory neurons and interneurons could play a critical role in the formation, stability, and maintenance of functional maps in the cortex.

On-line Adaptive Interval Type-2 Fuzzy Controller Design via Stable SPSA Learning Mechanism

This paper proposes an interval type-2 Takagi-Sugeno-Kang fuzzy neural system (IT2TFNS) to develop an on-line adaptive controller using stable simultaneous perturbation stochastic approximation (SPSA) algorithm. The proposed IT2TFNS realizes an interval type-2 TSK fuzzy logic system formed by the neural network structure. Differ from the most of interval type-2 fuzzy systems, the type-reduction of the proposed IT2TFNS is embedded in the network by using uncertainty bounds method such that the time-consuming Karnik-Mendel (KM) algorithm is replaced. The proposed stable SPSA algorithm provides the gradient free property and faster convergence. However, the stable SPSA algorithm inherently has the problem for on-line adaptive control. Hence, in order to achieve the on-line result, we utilize the sliding surface to develop a new on-line adaptive control scheme. In addition, the corresponding stable learning is derived by Lyapunov theorem which guarantees the convergence and stability of the closed-loop systems. Simulation and comparison results are shown to demonstrate the performance and effectiveness of our approach.

Modeling Component Concentrations of Sodium Aluminate Solution Via Hammerstein Recurrent Neural Networks

The component concentrations of sodium aluminate solution are important indices in alumina processing. At present, they are obtained by laboratory titration on samples taken from the production process. Due to the delays in taking and testing samples, they cannot be used for real-time control and optimization. Existing online measurements are not adopted because of the characteristics of the sodium aluminate solution such as high viscosity and the ease of precipitation which leads to pipeline blocking and decreased precision. In this paper, a new modeling method is proposed to measure the component concentrations online using the measurements of conductivity and temperature. The method combines the partial least squares (PLS) technique and the Hammerstein recurrent neural networks (HRNN), where a stable learning algorithm with theoretical analysis is given for the HRNN model. For this PLS-based HRNN, the PLS technique is used to solve the high dimensional and correlated data. Meanwhile, the HRNN technique is used to fit the nonlinear and dynamic characters of the process. An industrial experimental study on a sodium aluminate solution is described. The experiment results show that the proposed method is sufficient to warrant further evaluation in industrial scale experiments.

Two Types of Haar Wavelet Neural Networks for Nonlinear System Identification

Since wavelet transform uses the multi-scale (or multi-resolution) techniques for time series, wavelet transform is suitable for modeling complex signals. Haar wavelet transform is the most commonly used and the simplest one. The Haar wavelet neural network (HWNN) applies the Harr wavelet transform as active functions. It is easy for HWNN to model a nonlinear system at multiple time scales and sudden transitions. In this paper, two types of HWNN, feedforward and recurrent wavelet neural networks, are used to model discrete-time nonlinear systems, which are in the forms of the NARMAX model and state-space model. We first propose an optimal method to determine the structure of HWNN. Then two stable learning algorithms are given for the shifting and broadening coefficients of the wavelet functions. The stability of the identification procedures is proven.

Depression-Biased Reverse Plasticity Rule Is Required for Stable Learning at Top-Down Connections

Top-down synapses are ubiquitous throughout neocortex and play a central role in cognition, yet little is known about their development and specificity. During sensory experience, lower neocortical areas are activated before higher ones, causing top-down synapses to experience a preponderance of post-synaptic activity preceding pre-synaptic activity. This timing pattern is the opposite of that experienced by bottom-up synapses, which suggests that different versions of spike-timing dependent synaptic plasticity (STDP) rules may be required at top-down synapses. We consider a two-layer neural network model and investigate which STDP rules can lead to a distribution of top-down synaptic weights that is stable, diverse and avoids strong loops. We introduce a temporally reversed rule (rSTDP) where top-down synapses are potentiated if post-synaptic activity precedes pre-synaptic activity. Combining analytical work and integrate-and-fire simulations, we show that only depression-biased rSTDP (and not classical STDP) produces stable and diverse top-down weights. The conclusions did not change upon addition of homeostatic mechanisms, multiplicative STDP rules or weak external input to the top neurons. Our prediction for rSTDP at top-down synapses, which are distally located, is supported by recent neurophysiological evidence showing the existence of temporally reversed STDP in synapses that are distal to the post-synaptic cell body.

Evolutionarily stable learning schedules and cumulative culture in discrete generation models

Individual learning (e.g., trial-and-error) and social learning (e.g., imitation) are alternative ways of acquiring and expressing the appropriate phenotype in an environment. The optimal choice between using individual learning and/or social learning may be dictated by the life-stage or age of an organism. Of special interest is a learning schedule in which social learning precedes individual learning, because such a schedule is apparently a necessary condition for cumulative culture. Assuming two obligatory learning stages per discrete generation, we obtain the evolutionarily stable learning schedules for the three situations where the environment is constant, fluctuates between generations, or fluctuates within generations. During each learning stage, we assume that an organism may target the optimal phenotype in the current environment by individual learning, and/or the mature phenotype of the previous generation by oblique social learning. In the absence of exogenous costs to learning, the evolutionarily stable learning schedules are predicted to be either pure social learning followed by pure individual learning (“bang–bang” control) or pure individual learning at both stages (“flat” control). Moreover, we find for each situation that the evolutionarily stable learning schedule is also the one that optimizes the learned phenotype at equilibrium.