Eligibility Traces Research Articles

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze.

In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (αr) and the eligibility trace decay rate (λ), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman's correlation tests and linear regression. Our findings reveal that an αr of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for λ varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.

Time-scale invariant contingency yields one-shot reinforcement learning despite extremely long delays to reinforcement

Reinforcement learning inspires much theorizing in neuroscience, cognitive science, machine learning, and AI. A central question concerns the conditions that produce the perception of a contingency between an action and reinforcement-the assignment-of-credit problem. Contemporary models of associative and reinforcement learning do not leverage the temporal metrics (measured intervals). Our information-theoretic approach formalizes contingency by time-scale invariant temporal mutual information. It predicts that learning may proceed rapidly even with extremely long action-reinforcer delays. We show that rats can learn an action after a single reinforcement, even with a 16-min delay between the action and reinforcement (15-fold longer than any delay previously shown to support such learning). By leveraging metric temporal information, our solution obviates the need for windows of associability, exponentially decaying eligibility traces, microstimuli, or distributions over Bayesian belief states. Its three equations have no free parameters; they predict one-shot learning without iterative simulation.

Eligibility traces in an autonomous soccer robot with obstacle avoidance and navigation policy

As the thematic literature of machine learning suggests, reinforcement learning falls between the two methods of supervised learning and unsupervised learning. In this method, the learning agent receives a reward or punishment from the environment according to its action. Therefore, the learning agent interacts with the environment through trial and error and learns to choose the optimal action to achieve its goal. In the meantime, the eligibility traces are considered as one of the main mechanisms of reinforcement learning in receiving delayed rewards. In the use of conventional reinforcement learning methods, when the learning agent achieves a goal, only the value function of the last state-action pair changes, but all the trace states are affected based on the eligibility traces. In other words, the delayed rewards are distributed throughout the trace. Like the ant pheromone effect, this method can increase learning speed empirically to some extent. A soccer robot on the field encounters moving obstacles such as balls, rival robots, and home robots, and fixed obstacles such as gates and flags, so its environment is in a very dynamic state. Therefore, due to the dynamic environment, it is an important issue for autonomous soccer robots to avoid obstacles and should be considered in real play. The main idea of this study is to determine the appropriate traces for the robot to move towards the ball and ultimately to score with the approach of avoiding obstacles in simulating a real soccer match. The desired results obtained from a game played indicate a high level of online experience and decision-making power in the face of new situations.

An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning

The joint action-value function (JAVF) plays a key role in the centralized training of multi-agent deep reinforcement learning (MADRL)-based algorithms using the value function decomposition (VFD) and in the generating process of a collaborative policy between agents. However, under the influence of multiple factors such as environmental noise, inadequate exploration and iterative updating mechanism, estimation bias is inevitably introduced, causing its overestimation problem, which in turn prevents agents from obtaining accurate reward signals during the learning process, and fails to correctly approximate the optimal policy. To address this problem, this paper first analyzes the causes of joint action-value function overestimation, gives the corresponding mathematical proofs and theoretical derivations, and obtains the lower bound of the overestimation error; then, a MADRL overestimation reduction method based on the multi-step weighted double estimation named λWD QMIX is proposed. Specifically, the λWD QMIX method effectively achieves more stable and accurate JAVF estimation results using the bias correction estimation mechanisms based on the weighted double estimation and multi-step updating based on eligibility trace backup, without additionally adding or changing any network structure. The results of a series of experiments on the StarCraft II micromanipulation benchmark show that the proposed λWD QMIX algorithm can effectively improve the final performance and learning efficiency of the baseline algorithm, and can be seamlessly integrated with the partially MADRL algorithms based on communication learning.

From Past to Future: Rethinking Eligibility Traces

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the ????????????? ????? ????????. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD(λ)–a method that learns forward value functions, v^π, ????????. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation.

Distributed web hacking by adaptive consensus-based reinforcement learning

In this paper, we propose a novel adaptive consensus-based learning algorithm for automated and distributed web hacking. We aim to assist ethical hackers in conducting legitimate penetration testing and improving web security by identifying system vulnerabilities at an early stage. Ethical hacking is modeled as a capture-the-flag style task addressed within a distributed reinforcement learning framework. To achieve our goal, we employ interconnected intelligent agents that interact with their copies of the environment simultaneously to reach the target. They perform local information processing to optimize their policies and exchange information with neighboring agents. We propose a novel adaptive consensus scheme for inter-agent communications, which enables the agents to efficiently share network-wide information in a decentralized manner. The scheme dynamically adjusts its weights based on heuristics, involving both recency and frequency metrics of actions selected at a given state by an individual agent, similar to eligibility traces. We extensively analyze the convergence properties of our algorithm and introduce a new communication scheme design. We demonstrate that this design ensures the fastest convergence to the desired asymptotic values under the general setting of asymmetric communication topologies. Additionally, we provide a comprehensive review of the current state of the field and propose a web agent model with improved scalability compared to existing solutions. Numerical simulations are conducted to illustrate the key characteristics of our algorithm. The results demonstrate that it outperforms both non-cooperative and average consensus schemes. Moreover, our algorithm significantly reduces hacking times when compared to baseline algorithms that rely on more complex models. These findings offer valuable insights to system security administrators, enabling them to address identified shortcomings and vulnerabilities effectively.

A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation

The stochastic approximation (SA) method stands as the foundational mathematical tool for modern large-scale optimization and machine learning. Therefore, gaining a theoretical understanding of SA algorithms is of fundamental interest. In their paper titled “A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation,” Chen et al. present a unified Lyapunov framework for the finite-sample analysis of a Markovian SA algorithm under a contractive operator with respect to an arbitrary norm. The key novelty lies in the construction of a smooth Lyapunov function called the generalized Moreau envelope. The authors demonstrate the effectiveness of their SA results in the context of reinforcement learning (RL), specifically through popular algorithms such as variants of temporal difference (TD) learning and Q-learning. As byproducts, the results provide theoretical insights into the efficiency of bootstrapping in TD learning with eligibility traces and the bias-variance tradeoff in off-policy learning.

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Spatiotemporal motor learning with reward-modulated Hebbian plasticity in modular reservoir computing

Generation of complex patterns at a specific timing is crucial to most forms of learning and behavior, which are acquired through dopamine-modulated plasticity in the striatum. However, the neural mechanisms of such reward-based spatiotemporal processing remain unknown. Inspired by the cortico-striatal circuits, this study developed a new reservoir computing method, a class of recurrent neural networks, based on reward-modulated Hebbian learning (RMHL) for the spatiotemporal motor learning. We utilized a reservoir of basal dynamics (reBASICS), which generated self-sustained limit cycle oscillations with various frequencies, as a reservoir structure. Then, the oscillations were linearly integrated as readout output, in which readout weights were modulated with RMHL. The simulations showed that reBASICS-based RMHL was able to accomplish both motor timing and pattern drawing tasks, for which existing reservoir-based RMHL failed. Further, introducing an eligibility trace mechanism into RMHL allowed the model to learn motor timing even when reward-based modulation was delayed. In conclusion, this model is proposed as a new computational model of temporal processing of the striatum, where the cortical areas generate stable oscillations. From the oscillatory dynamics, spatiotemporal patterns are learned using RMHL in the striatum.

Variational eligibility trace meta-reinforcement recurrent network for residual life prediction of space rolling bearings

Traditional sequence recurrent neural networks (SRNNs) have the defect of long time dependence in the prediction of time series, resulting in their poor generalization ability. Moreover, it is required to traverse the whole training data set to realize supervised learning by SRNNs, which increases the time complexity and leads to their low prediction accuracy and high computation cost in the residual life prediction of space rolling bearings in the ground simulated space environment. In view of this, a novel SRNN named variational eligibility trace meta-reinforcement recurrent network (VETMRRN) is proposed for achieving higher residual life prediction accuracy and lower computation cost. In the proposed VETMRRN, a new sequence recurrent network structure is constructed to increase the memory amount of historical information, thus improving the long-term memory capacity of VETMRRN. Then, a hyperparameter self-initialization meta-learning network with an oracle gate mechanism is designed to self-initialize the hyperparameters of VETMRRN for fast determination of the optimal review sequence length. Hence, VETMRRN can adapt to different input sequence lengths and avoid the defect of long time dependence of traditional SRNNs. Furthermore, a variational auto-encoding meta policy gradient learning algorithm with an eligibility trace operator is designed to improve the training speed and enhance the global optimization effect for VETMRRN parameters. Based on the above advantages of VETMRRN, a new residual life prediction method of space rolling bearings in the ground simulated space environment is proposed. Firstly, the time-frequency fusion features are extracted by Shapely-value feature fusion from the vibration acceleration data of space rolling bearing as the performance degradation features. Then, the performance degradation features are input into VETMRRN to predict the performance degradation feature trends of space rolling bearings. Finally, a Weibull-distribution reliability model is established based on the performance degradation feature trend values to predict the residual life of space rolling bearings. The effectiveness of the proposed VETMRRN-based prediction method is verified by the vibration acceleration data collected from the self-built vibration monitoring platform of space rolling bearings in the ground simulated space environment. The results indicate that compared to traditional SRNNs, deep sparse auto-encoding neural network (DSAE-NN), and multi-kernel least-square support vector machine (MK-LSSVM), the proposed method can improve the prediction accuracy and reduce the computation cost in the residual life prediction of space rolling bearings. In the future, the generalization performance of VETMRRN still needs to be further improved.

State-transition-free reinforcement learning in chimpanzees (Pan troglodytes).

The outcome of an action often occurs after a delay. One solution for learning appropriate actions from delayed outcomes is to rely on a chain of state transitions. Another solution, which does not rest on state transitions, is to use an eligibility trace (ET) that directly bridges a current outcome and multiple past actions via transient memories. Previous studies revealed that humans (Homo sapiens) learned appropriate actions in a behavioral task in which solutions based on the ET were effective but transition-based solutions were ineffective. This suggests that ET may be used in human learning systems. However, no studies have examined nonhuman animals with an equivalent behavioral task. We designed a task for nonhuman animals following a previous human study. In each trial, participants chose one of two stimuli that were randomly selected from three stimulus types: a stimulus associated with a food reward delivered immediately, a stimulus associated with a reward delivered after a few trials, and a stimulus associated with no reward. The presented stimuli did not vary according to the participants' choices. To maximize the total reward, participants had to learn the value of the stimulus associated with a delayed reward. Five chimpanzees (Pan troglodytes) performed the task using a touchscreen. Two chimpanzees were able to learn successfully, indicating that learning mechanisms that do not depend on state transitions were involved in the learning processes. The current study extends previous ET research by proposing a behavioral task and providing empirical data from chimpanzees.

A citywide TD‐learning based intelligent traffic signal control for autonomous vehicles: Performance evaluation using SUMO

AbstractAn autonomous vehicle can sense its environment and operate without human involvement. Its adequate management in an intelligent transportation system could significantly reduce traffic congestion and overall travel time in a network. Adaptive traffic signal controller (ATSC) based on multi‐agent systems using state‐action‐reward‐state‐action (SARSA ()) are well‐known state‐of‐the‐art models to manage autonomous vehicles within urban areas. However, this study found inefficient weights updating mechanisms of the conventional SARSA () models. Therefore, it proposes a Gaussian function to regulate the eligibility trace vector's decay mechanism effectively. On the other hand, an efficient understanding of the state of the traffic environment is crucial for an agent to take optimal actions. The conventional models feed the state values to the agents through the MinMax normalization technique, which sometimes shows less efficiency and robustness. So, this study suggests the MaxAbs scaled state values instead of MinMax to address the problem. Furthermore, the combination of the A‐star routing algorithm and proposed model demonstrated a good increase in performance relatively to the conventional SARSA ()‐based routing algorithms. The proposed model and the baselines were implemented in a microscopic traffic simulation environment using the SUMO package over a complex real‐world‐like ‐intersections network to evaluate their performance. The results showed a reduction of the vehicle's average total waiting time and total stops by a mean value of % and % compared to the considered baselines. Also, the A‐star combined with the proposed controller outperformed the conventional approaches by increasing the vehicle's average trip speed by %.

Event-based online learning control design with eligibility trace for discrete-time unknown nonlinear systems

As a heuristic algorithm to solve the nonlinear optimal control problem, adaptive dynamic programming is commonly constructed from one-step temporal difference learning. Eligibility trace can effectively speed up controller learning by considering the multi-step information in reinforcement learning. However, eligibility trace will bring in additional computational consumption and increase the learning burden. This paper attempts to take advantage of eligibility trace and avoids the high learning consumption at the same time. Therefore, an event-based neural dynamic programming (λ) [ENDP(λ)] algorithm via the actor–critic framework is constructed to solve the near-optimal control problem of unknown discrete-time systems. First, the modified forward view with eligibility trace is derived, which is suitable for engineering practice. Second, based on the event-triggered mechanism, ENDP(λ) is designed to relieve the pressure of communication consumption. Then, the event-based system is proven to ensure the input-to-state stability under a suitable triggering condition. Moreover, three neural networks are given to approximate the one-step cost function, the n-step cost function, and the control law, respectively. Finally, two typical experimental simulation examples are presented to verify the effectiveness of the ENDP(λ) algorithm.

Presynaptic spike-driven plasticity based on eligibility trace for on-chip learning system.

Recurrent spiking neural network (RSNN) performs excellently in spatio-temporal learning with backpropagation through time (BPTT) algorithm. But the requirement of computation and memory in BPTT makes it hard to realize an on-chip learning system based on RSNN. In this paper, we aim to realize a high-efficient RSNN learning system on field programmable gate array (FPGA). A presynaptic spike-driven plasticity architecture based on eligibility trace is implemented to reduce the resource consumption. The RSNN with leaky integrate-and-fire (LIF) and adaptive LIF (ALIF) models is implemented on FPGA based on presynaptic spike-driven architecture. In this architecture, the eligibility trace gated by a learning signal is used to optimize synaptic weights without unfolding the network through time. When a presynaptic spike occurs, the eligibility trace is calculated based on its latest timestamp and drives synapses to update their weights. Only the latest timestamps of presynaptic spikes are required to be stored in buffers to calculate eligibility traces. We show the implementation of this architecture on FPGA and test it with two experiments. With the presynaptic spike-driven architecture, the resource consumptions, including look-up tables (LUTs) and registers, and dynamic power consumption of synaptic modules in the on-chip learning system are greatly reduced. The experiment results and compilation results show that the buffer size of the on-chip learning system is reduced and the RSNNs implemented on FPGA exhibit high efficiency in resources and energy while accurately solving tasks. This study provides a solution to the problem of data congestion in the buffer of large-scale learning systems.

Proposal of Deterministic Policy Gradient Method for Port-Hamiltonian Systems Using Eligibility Trace and Verification by Numerical Experiment

This paper proposes a deterministic policy gradient method for port-Hamiltonian systems using an eligibility trace. The deterministic policy gradient method commonly uses one of two types of algorithms, either the on- or off-policy method. The proposed algorithm employs the off-policy method to perform a probabilistic search. In addition, we introduce an eligibility trace to the method to speed up the learning process. A numerical simulation shows the effectiveness of the proposed method.

A second-order dynamic and static ship path planning model based on reinforcement learning and heuristic search algorithms

Ship path planning plays an important role in the intelligent decision-making system which can provide important navigation information for ship and coordinate with other ships via wireless networks. However, existing methods still suffer from slow path planning and low security problems. In this paper, we propose a second-order ship path planning model, which consists of two main steps, i.e., first-order static global path planning and second-order dynamic local path planning. Specifically, we first create a raster map using ArcGIS. Second, the global path planning is performed on the raster map based on the Dyna-Sarsa(lambda) model, which integrates the eligibility trace and the Dyna framework on the Sarsa algorithm. Particularly, the eligibility trace has a short-term memory for the trajectory, which can improve the convergence speed of the model. Meanwhile, the Dyna framework obtains simulation experience through simulation training, which can further improve the convergence speed of the model. Then, the improved ship trajectory prediction model based on stacked bidirectional gated recurrent unit is used to identify the risk of ship collision and switch the path planning from the first order to the second order. Finally, the second-order dynamic local path planning is presented based on the FCC-A* algorithm, where the cost function of the traditional path planning A* algorithm is rewritten using the fuzzy collision cost membership function (fuzzy collision cost, FCC) to reduce the collision risk of ships. The proposed model is evaluated on the Baltic Sea geographic information and ship trajectory datasets. The experimental results show that the eligibility trace and the Dyna learning framework in the proposed model can effectively improve the planning efficiency of the ship’s global path planning, and the collision risk membership function can effectively reduce the number of collisions in A* local path planning and thus improve the navigation safety of encountering ships.

A robotic model of hippocampal reverse replay for reinforcement learning.

Hippocampal reverse replay, a phenomenon in which recently active hippocampal cells reactivate in the reverse order, is thought to contribute to learning, particularly reinforcement learning (RL), in animals. Here, we present a novel computational model which exploits reverse replay to improve stability and performance on a homing task. The model takes inspiration from the hippocampal-striatal network, and learning occurs via a three-factor RL rule. To augment this model with hippocampal reverse replay, we derived a policy gradient learning rule that associates place-cell activity with responses in cells representing actions and a supervised learning rule of the same form, interpreting the replay activity as a 'target' frequency. We evaluated the model using a simulated robot spatial navigation task inspired by the Morris water maze. Results suggest that reverse replay can improve performance stability over multiple trials. Our model exploits reverse reply as an additional source for propagating information about desirable synaptic changes, reducing the requirements for long-time scales in eligibility traces combined with low learning rates. We conclude that reverse replay can positively contribute to RL, although less stable learning is possible in its absence. Analogously, we postulate that reverse replay may enhance RL in the mammalian hippocampal-striatal system rather than provide its core mechanism.

Frequency matching optimization model of ultrasonic scalpel transducer based on neural network and reinforcement learning

Aiming at the problem that the excitation frequency and resonant frequency of the transducer cannot keep synchronous, the output amplitude decreases and the vibration is unstable. In this study, the working principle of piezoelectric transducers is firstly analyzed by the equivalent circuit method and the instantaneous characteristic variables (installing preload, assembly preload, load, and other factors) that affect the frequency matching obtained to establish the equivalent relationship with the resonant frequency. Secondly, in order to make the synchronization between excitation frequency and resonance frequency, the key instantaneous characteristic variables are extracted based on Pearson correlation coefficient. Thirdly, the mathematical model of the mapping relationships between instantaneous characteristic variables and resonance frequency is established with the radial basis function neural network (RBFNN). Fourthly, for the purpose of the adaptation to the characteristics of dynamic load and real-time frequency modulation in the operation of ultrasonic scalpels, the reinforcement learning (Q-learning algorithm) and the weight vector of RBFNN are used to define the eligibility trace, which is used to dynamically adjust the RBFNN frequency matching optimization model in real time and maintain the “constant” amplitude output and stable harmonic response process. Finally, the experimental results show that, compared with the traditional methods, the frequency matching optimization model of ultrasonic scalpel transducer based on RBF neural network and Q-Learning reinforcement learning is effective, and the vibration amplitude of the transducer is increased by 15.25μm. The amplitude fluctuation is stable at 0.92μm. It can provide decision-making guidance for relevant engineering fields.

Postsynaptic burst reactivation of hippocampal neurons enables associative plasticity of temporally discontiguous inputs.

A fundamental unresolved problem in neuroscience is how the brain associates in memory events that are separated in time. Here, we propose that reactivation-induced synaptic plasticity can solve this problem. Previously, we reported that the reinforcement signal dopamine converts hippocampal spike timing-dependent depression into potentiation during continued synaptic activity (Brzosko et al., 2015). Here, we report that postsynaptic bursts in the presence of dopamine produce input-specific LTP in mouse hippocampal synapses 10 min after they were primed with coincident pre- and post-synaptic activity (post-before-pre pairing; Δt = -20 ms). This priming activity induces synaptic depression and sets an NMDA receptor-dependent silent eligibility trace which, through the cAMP-PKA cascade, is rapidly converted into protein synthesis-dependent synaptic potentiation, mediated by a signaling pathway distinct from that of conventional LTP. This synaptic learning rule was incorporated into a computational model, and we found that it adds specificity to reinforcement learning by controlling memory allocation and enabling both 'instructive' and 'supervised' reinforcement learning. We predicted that this mechanism would make reactivated neurons activate more strongly and carry more spatial information than non-reactivated cells, which was confirmed in freely moving mice performing a reward-based navigation task.

Battery life constrained real-time energy management strategy for hybrid electric vehicles based on reinforcement learning

Hybrid electric vehicles (HEVs) bridge the gap between internal combustion engine vehicles and pure electric vehicles, and are therefore regarded as a promising solution to the energy crisis. This paper proposes a real-time energy management strategy (EMS) for hybrid electric vehicles based on reinforcement learning (RL) to improve fuel economy and minimize battery degradation. First, an online recursive Markov chain (MC) is developed that continuously collects statistical features from actual driving conditions, and thus an adaptive and accurate environment model is established. Then, a novel RL algorithm, eligibility trace, is introduced to learn the control policy online based on MC model. By introducing a trace-decay parameter, the eligibility trace algorithm unifies the returns of different steps, forming a more reliable estimate of the optimal value function, and therefore outperforms traditional RL algorithms in optimization. Furthermore, induced matrix norm (IMN) is employed as a standard to measure difference between transition probability matrices (TPM) of MC and to decide when to update environment model as well as recalculate the control policy. Therefore, the EMS's adaptability to various driving conditions are significantly enhanced. Simulation results indicate that eligibility trace shows the best performance in both improving fuel economy and reducing battery life loss compared with Q-learning and rule-based method.